Source-linked AI summary
Context Encoding for Semantic Segmentation
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, Amit Agrawal
TL;DR
Semantic segmentation methods have improved spatial resolution, but the paper asks how explicit global context can improve pixelwise labeling. It introduces a Context Encoding Module that captures scene semantics and selectively highlights class-dependent featuremaps, achieving strong segmentation results and improving shallow-network classification.
Problem
Existing segmentation approaches improve resolution and receptive fields, but the paper identifies limited explicit use of global scene context as an open question.
Method
The Context Encoding Module captures global feature statistics, predicts scaling factors, and selectively emphasizes or de-emphasizes class-dependent featuremaps; Semantic Encoding Loss regularizes scene-level category prediction.
Results
The approach improves semantic segmentation across PASCAL-Context, PASCAL VOC 2012, and ADE20K, while also improving representations in shallow CIFAR-10 networks.
Takeaways & Limitations
Context Encoding is lightweight and compatible with existing FCN-based approaches, with experiments showing improved feature representations using global context.
Abstract
from arXiv · showhide
Recent work has made significant progress in improving spatial resolution for pixelwise labeling with Fully Convolutional Network (FCN) framework by employing Dilated/Atrous convolution, utilizing multi-scale features and refining boundaries. In this paper, we explore the impact of global contextual information in semantic segmentation by introducing the Context Encoding Module, which captures the semantic context of scenes and selectively highlights class-dependent featuremaps. The proposed Context Encoding Module significantly improves semantic segmentation results with only marginal extra computation cost over FCN. Our approach has achieved new state-of-the-art results 51.7% mIoU on PASCAL-Context, 85.9% mIoU on PASCAL VOC 2012. Our single model achieves a final score of 0.5567 on ADE20K test set, which surpass the winning entry of COCO-Place Challenge in 2017. In addition, we also explore how the Context Encoding Module can improve the feature representation of relatively shallow networks for the image classification on CIFAR-10 dataset. Our 14 layer network has achieved an error rate of 3.45%, which is comparable with state-of-the-art approaches with over 10 times more layers. The source code for the complete system are publicly available.
1. Introduction
The paper argues that semantic segmentation needs explicit global scene context, not only larger receptive fields or multi-scale features, and introduces a context-aware module to selectively emphasize class-dependent featuremaps.
- Semantic segmentation assigns object-category predictions to every pixel, describing each object’s category, location, and shape.
- Dilated convolution preserves spatial resolution but can isolate pixels from global scene context, causing misclassifications such as windowpanes labeled as doors.
- Multi-scale pyramid methods enlarge receptive fields and improve performance, but their context representations remain implicit.
- Classic visual encoders capture global context through feature statistics, motivating explicit contextual representations in modern segmentation networks.
- The Context Encoding Module captures global context and selectively highlights class-dependent featuremaps, while Semantic Encoding Loss encourages prediction of object-category presence.
- EncNet combines the Context Encoding Module with a pre-trained ResNet and dilation strategy as a semantic segmentation framework.
2. Context Encoding Module
The Context Encoding Module captures global semantic context, uses it to rescale class-dependent featuremaps, and regularizes learning with semantic presence prediction. EncNet integrates this module into a dilated FCN pipeline with marginal additional computation.
- Context Encoding: The Encoding Layer captures global semantic context from convolutional featuremaps using learned codewords and aggregated residual encoders.For an input featuremap of shape C × H × W, spatial features are represented as C-dimensional inputs and aggregated across K codewords.
- Featuremap Attention: Featuremap Attention predicts scaling factors from encoded semantics to selectively emphasize or de-emphasize class-dependent featuremaps.A fully connected layer with sigmoid activation produces the factors, which are applied through channel-wise multiplication.
- Semantic Encoding Loss: Semantic Encoding Loss regularizes the module by requiring prediction of object-category presence in the scene alongside per-pixel segmentation.The separate SE-loss branch uses encoded semantics as input, while the segmentation branch produces per-pixel predictions.
- Context Encoding Network: EncNet augments a pre-trained ResNet with the Context Encoding Module and uses dilation at stages 3 and 4.The module is placed before final prediction, and SE-loss branches are added to the base network at both dilated stages.
- Relation to Other Approaches: The differentiable module inserts into existing FCN pipelines without extra supervision or framework changes, adding only marginal computation.The framework is designed to leverage global context while retaining compatibility with FCN-based approaches.
3. Experimental Results
Experiments evaluate EncNet across semantic segmentation benchmarks and CIFAR-10 classification, including ablations of SE-loss, codewords, architecture depth, and evaluation settings.
- Ablation Study: α = 0.2 gives the best SE-loss performance, while improvements saturate at K = 32 codewords rather than K = 0 global average pooling.These settings are selected from the Figure 5 ablations using single-scale evaluation.
- PASCAL-Context: 52.6% mIoU and 81.2% pixAcc are EncNet’s final PASCAL-Context scores, with 51.7% mIoU when background is included.The final scores use multisize evaluation; ResNet101 adds 2.5% mIoU over the shallower network.
- PASCAL VOC 2012: 85.9% mIoU is EncNet’s best PASCAL VOC 2012 result with COCO pre-training, compared with 82.9% without COCO data.The paper reports lower computation complexity than PSPNet and DeepLabv3 in this comparison.
- ADE20K: 0.55675 is EncNet’s ADE20K test score, surpassing PSP-Net-269 and all entries in the COCO-Place Challenge 2017.EncNet-101 is described as comparable with PSPNet-269 on the ADE20K validation benchmark.
- CIFAR-10 Classification: 3.45% error is achieved by the 14-layer Context Encoding network on CIFAR-10, comparable with state-of-the-art approaches using much deeper networks.The experiments also report that the training process likely improves all approaches.
4. Conclusion
The paper concludes that Context Encoding captures scene context to selectively highlight class-dependent featuremaps while remaining lightweight and compatible with FCN approaches.
- 4. Conclusion: The Context Encoding Module selectively highlights class-dependent featuremaps and “simplifies” the semantic segmentation problem for the network.The conclusion characterizes the module as conceptually straightforward, light-weight, and compatible with existing FCN base approaches.
A. Implementation Details on Synchronized Cross-GPU Batch Normalization
The implementation computes synchronized batch-normalization statistics across devices using globally reduced sums, requiring one synchronization for forward statistics.
- Forward Statistics: The variance is represented from the sums of inputs and squared inputs, avoiding separate synchronization for mean and variance.The method is presented as requiring one synchronization in each iteration.
- Forward Statistics: The implementation computes global sums of xi and x_i^2 with an all-reduce operation after per-device calculation.Global mean and variance are then used for normalization of each sample.
- Backward Statistics: Back-propagation likewise synchronizes once for the gradients of the summed xi and x_i^2 statistics.The normalization uses yi = γ(xi−µ)/sqrt(σ^2+ϵ) + β.