Source-linked AI summary
ParseNet: Looking Wider to See Better
Wei Liu, Andrew Rabinovich, Alexander C. Berg
TL;DR
Semantic segmentation models based on FCNs can miss useful scene-level context because they process local regions independently. ParseNet adds a pooled whole-image feature to local features within an end-to-end network, with learned normalization for feature combination. The method improves FCN baselines with small computational overhead, reaches state-of-the-art results on SiftFlow and PASCAL-Context, and approaches state of the art on PASCAL VOC2012.
Problem
FCN-based segmentation processes local regions independently and disregards global image information that could provide useful scene-level semantic context.
Method
ParseNet pools a feature map over the whole image, appends the resulting context feature to local features, and uses normalized feature combinations with learned scaling.
Results
ParseNet achieves state-of-the-art performance on SiftFlow and PASCAL-Context, near state-of-the-art performance on PASCAL VOC2012, and improves FCN baselines with small computational overhead.
Takeaways & Limitations
Global features provide a simple, robust way to improve FCN performance by incorporating contextual information, while remaining compatible with explicit structure prediction.
Takeaways & Limitations
Spatial-pyramid and lower-level subregion features did not produce significant improvements, which the authors associate with already-large receptive fields in high-level feature maps.
Abstract
from arXiv · showhide
We present a technique for adding global context to deep convolutional networks for semantic segmentation. The approach is simple, using the average feature for a layer to augment the features at each location. In addition, we study several idiosyncrasies of training, significantly increasing the performance of baseline networks (e.g. from FCN). When we add our proposed global feature, and a technique for learning normalization parameters, accuracy increases consistently even over our improved versions of the baselines. Our proposed approach, ParseNet, achieves state-of-the-art performance on SiftFlow and PASCAL-Context with small additional computational cost over baselines, and near current state-of-the-art performance on PASCAL VOC 2012 semantic segmentation with a simple approach. Code is available at https://github.com/weiliu89/caffe/tree/fcn .
1 INTRODUCTION
ParseNet addresses FCN’s lack of global image information by injecting pooled global context into an end-to-end fully convolutional network. The approach improves segmentation performance with small computational overhead and can reach accuracy comparable to structure-based post-processing.
- Motivation: FCN processes sliding-window regions independently, disregarding global image information that could provide useful scene-level semantic context.This limits the pixel distance across which labeling consistency is maintained.
- Approach: ParseNet integrates global context into an end-to-end fully convolutional network that jointly predicts all pixel values instead of dividing images into regions.The method contrasts with patch-based approaches and adds only small computational overhead.
- Approach: A feature map is pooled over the whole image into a context vector, unpooled, and appended to the standard feature map at a subsequent layer.The operation can be applied selectively or to multiple feature maps.
- Approach: L2 normalization and a scaling factor learned through backpropagation address differences in feature scale when combining layers.These learned normalization parameters are part of the proposed feature-combination strategy.
- Results: The global-context operations significantly improve basic FCN performance, reaching accuracy on par with Chen et al.’s structure-based post-processing method.The network can also be combined with explicit structure prediction such as a CRF.
2 RELATED WORK
Related work establishes that contextual information helps semantic segmentation but often relies on patch-based features or decoupled graphical-model post-processing. ParseNet instead uses global pooling within the FCN framework, motivated by the gap between theoretical and empirical receptive fields.
- Contextual segmentation: Context from the whole image can improve local patch classification, and prior methods concatenate global features or use multi-scale “zoom-out” features.These approaches pool features differently for local patches and the whole image, complicating end-to-end training.
- Receptive fields: The theoretical receptive field of a deep network can be large while its empirical receptive field remains insufficient to capture global context.Figure 2 distinguishes the image, activation map, theoretical receptive field, and empirical receptive field.
- ParseNet: ParseNet applies global average pooling to the final or another feature map and uses the resulting whole-image feature as context within FCN.Experiments report improved local patch prediction after adding this context.
- Graphical models: FCN- and CNN-based graphical-model methods combine unary predictions with spatial smoothing, but decoupled stages make optimization of the final segmentation objective difficult.This motivates more unified approaches that incorporate context and smoothness during learning.
3 PARSENET
ParseNet adds whole-image context to local feature maps and uses normalization with learned channel-wise scaling to combine features robustly. These choices improve semantic segmentation while addressing scale differences across layers.
- Global Context: Whole-image context improves ambiguous per-pixel classification, whereas finer spatial-pyramid pooling did not provide significant improvements.The authors conjecture that high-level feature maps have empirical receptive fields larger than or similar to the pooled subregions.
- Global Context: Global average pooling produces a context vector that is replicated spatially and appended to local feature maps.The global feature can be fused early by concatenation or later through separate classifiers whose predictions are merged.
- L2 Normalization Layer: Naively concatenated features can be dominated by larger-scale activations, making the combined representation less discriminative and training sensitive to parameter tuning.Features from different layers can differ substantially in scale, with some activations roughly 2 orders of magnitude smaller than others.
- L2 Normalization Layer: L2 normalization followed by learned channel-wise scaling stabilizes feature combination and improves performance without requiring cross-sample aggregation.The scaling parameter γ_i restores an appropriate magnitude for each normalized channel, while the extra parameters equal the total number of channels.
- L2 Normalization Layer: The normalization is applied per pixel in a feature map, extending the feature-vector equations elementwise.This operation supports both feature combination and late fusion involving lower-level features with large norms.
4 EXPERIMENTS
Experiments across SiftFlow, PASCAL-Context, and PASCAL VOC2012 evaluate finetuning, feature fusion, global context, and normalization. Global context and learned feature scaling improve results in several settings, while its usefulness depends on dataset characteristics and feature receptive fields.
- PASCAL VOC2012: 5% improvement raises reproduced DeepLab performance on VOC2012 from 59.80 to 64.96 using parameters found on PASCAL-Context.DeepLab-LargeFOV also gains 3.5% with the same parameters.
- SiftFlow: SiftFlow contains 2,688 images and 33 semantic categories, and adding more feature layers generally improves performance.Global context provides little additional benefit because fc7’s receptive field is similar to or larger than the 256 × 256 images.
- PASCAL-Context: 1.6% improvement on PASCAL-Context results from adding global context, while L2 normalization prevents performance drops when combining additional layers.Without normalization, adding conv4 slightly decreases performance and adding conv3 collapses the network; normalization enables state-of-the-art performance as reported.
- PASCAL VOC2012: 67.49% versus 64.92% shows that adding normalized pool6 improves the DeepLab VOC2012 baseline.Without normalizing fc7 and pool6, the experiment reports no improvement.
- PASCAL VOC2012: 3.8% improvement follows adding pool6 to the ParseNet VOC2012 baseline, with early and late fusion performing similarly.Learning scaling factors after normalization is important; without normalization and learned scaling, the effect is diminished.
- PASCAL VOC2012: Global context can correct local confusions but can also bias predictions toward visually similar categories, such as classifying a spotty horse as a dog.Learning weights for pool6 and fc7 after L2 normalization improves performance overall.
- PASCAL VOC2012: Lower-level features and 1 × 1, 2 × 2, and 4 × 4 pooled subregions yield no significant VOC2012 improvement with either early or late fusion.The authors conjecture that high-level features already have sufficiently large receptive fields and that lower-level features are not sufficiently discriminative.
- PASCAL VOC2012: VOC2012 test performance with global context is within the standard deviation of Chen et al. (2014), which uses a fully connected CRF for output smoothing.The baseline already exceeds many existing methods, which the authors attribute to proper finetuning.
5 CONCLUSION
ParseNet directly incorporates global context into fully convolutional semantic segmentation, while emphasizing simple training and robust performance across benchmarks.
- ParseNet directly includes global context in a fully convolutional architecture for semantic segmentation.
- The largest receptive field of an FCN does not provide sufficient global context, so global context must be modeled directly.
- On PASCAL VOC2012, ParseNet performs within the standard deviation of DeepLab-LargeFOV-CRF.
- ParseNet reaches state-of-the-art results on SiftFlow and PASCAL-Context and near state-of-the-art results on PASCAL VOC2012.
- The approach is designed for simplicity and robustness of learning, with results presented on three benchmark datasets.