Source-linked AI summary
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing
Qibin Hou, Li Zhang, Ming-Ming Cheng, Jiashi Feng
TL;DR
Scene-parsing networks need broader contextual modeling because local convolutional and pooling operators provide limited effective fields-of-view, while square windows inadequately capture anisotropic context. The paper introduces long, narrow strip pooling and builds a Strip Pooling Module and Mixed Pooling module that integrate into existing backbones. The resulting SPNet achieves significant improvements over baselines and new state-of-the-art results on widely used scene-parsing benchmarks.
Problem
Local convolutional and pooling operators limit effective fields-of-view, while square-window methods restrict the modeling of anisotropic context in complex scenes.
Method
The paper introduces strip pooling with 1 × N or N × 1 kernels and develops lightweight Strip Pooling and Mixed Pooling modules for existing backbones.
Results
SPNet achieves significant improvements over baselines and establishes new state-of-the-art results on widely used scene-parsing benchmark datasets.
Takeaways & Limitations
Strip-based pooling provides a lightweight, plug-and-play design for aggregating long-range context while retaining local details in scene parsing networks.
Abstract
from arXiv · showhide
Spatial pooling has been proven highly effective in capturing long-range contextual information for pixel-wise prediction tasks, such as scene parsing. In this paper, beyond conventional spatial pooling that usually has a regular shape of NxN, we rethink the formulation of spatial pooling by introducing a new pooling strategy, called strip pooling, which considers a long but narrow kernel, i.e., 1xN or Nx1. Based on strip pooling, we further investigate spatial pooling architecture design by 1) introducing a new strip pooling module that enables backbone networks to efficiently model long-range dependencies, 2) presenting a novel building block with diverse spatial pooling as a core, and 3) systematically comparing the performance of the proposed strip pooling and conventional spatial pooling techniques. Both novel pooling-based designs are lightweight and can serve as an efficient plug-and-play module in existing scene parsing networks. Extensive experiments on popular benchmarks (e.g., ADE20K and Cityscapes) demonstrate that our simple approach establishes new state-of-the-art results. Code is made available at https://github.com/Andrew-Qibin/SPNet.
1. Introduction
Conventional local pooling struggles with long-range and anisotropic context in complex scenes. The paper introduces lightweight strip pooling and two pooling-based modules to combine long-range and local context for scene parsing.
- FCN-based scene parsing methods rely mainly on local convolutions and pooling, limiting their effective fields-of-view in complex scenes.
- Square-window context methods can miss anisotropic structures and may include irrelevant regions when objects have irregular shapes.
- Strip pooling uses a long kernel along one spatial dimension and a narrow kernel along the other to capture global relations while preserving local context.
- The Strip Pooling Module encodes long-range context separately along horizontal and vertical dimensions for feature refinement.
- The Mixed Pooling module uses different pooling kernel shapes to model long-range dependencies at high semantic levels, while SPNet integrates both modules into a ResNet backbone.
- Both modules are lightweight add-on blocks, and the resulting architecture reports significant baseline improvements and new state-of-the-art results on scene parsing benchmarks.
2. Related Work
Prior scene-parsing methods expand receptive fields through multi-scale, dilated, non-local, or pyramid pooling approaches. The paper identifies square kernels and limited modularity as shortcomings addressed by strip-based modules.
- Multi-scale feature aggregation fuses contextual information using filters or pooling operations at multiple rates and fields-of-view.
- Dilated convolutions widen receptive fields without extra parameters, while non-local methods model long-range context in continuous feature space.
- Spatial pyramid pooling captures large-range context with parallel pooling operations, but its square kernels limit contextual flexibility.
- Unlike spatial pyramid pooling, the proposed strip and mixed pooling modules use 1 × N or N × 1 kernels and can be plugged into and stacked within existing networks.
- The paper reports that its modular strip-based designs are more powerful and adaptable than spatial pyramid pooling in experiments.
3. Methodology
The methodology introduces strip pooling as a long-range context mechanism and builds lightweight SPM and MPM modules that augment backbone networks for scene parsing.
- 3.1. Strip Pooling: Strip pooling uses band-shaped windows with spatial extents (H, 1) or (1, W) to pool along one spatial dimension.Unlike square pooling, it averages feature values across a row or column.
- 3.1. Strip Pooling: Horizontal and vertical strip pooling capture long-range dependencies across distributed regions while retaining local details along the narrow dimension.The long, narrow kernels distinguish strip pooling from conventional square-kernel pooling.
- 3.2. Strip Pooling Module: The Strip Pooling Module combines parallel horizontal and vertical pathways with 1D convolutions to gather long-range context for backbone feature refinement.Each pathway applies strip pooling followed by a kernel-size-3 1D convolution.
- 3.2. Strip Pooling Module: The SPM output uses element-wise scaling by a sigmoid-transformed 1 × 1 convolution of fused strip-pooled features, keeping the module lightweight.The paper selects this combination for efficiency and reports that it works well.
- 3.3. Mixed Pooling Module: The Mixed Pooling Module combines short-range and long-range dependency aggregation through diverse pooling operations in bottleneck residual blocks.Its long-range sub-module uses horizontal and vertical strip pooling, while spatial pooling addresses closely distributed semantic regions.
- 3.4. Overall Architecture: SPNet integrates SPMs and MPMs with dilated ResNet backbones, adding two MPMs after reducing backbone output channels from 2048 to 1024.Each MPM uses 256-channel 3 × 3 or 3 convolutions, followed by a convolutional segmentation-map predictor.
4. Experiments
Experiments evaluate strip-pooling modules and their configurations across scene-parsing benchmarks, with ablations testing module count, dependency ranges, placement, pooling type, and backbone settings.
- Ablation Studies: Adding two MPMs yields around a 4.3% performance gain, whereas additional MPMs provide trivial gains; the default is therefore two MPMs.The authors attribute diminishing returns to an already sufficiently large receptive field and consider runtime cost.
- Ablation Studies: Base FCN plus two MPMs outperforms PSPNet despite using 12M fewer parameters.The comparison is reported as evidence for the modularized MPM design over PPM.
- Ablation Studies: SPM placement raises base-FCN mIoU from 37.63% to 41.66%, while the default combined placement reaches 44.03%.The default places SPMs in the last building block of each stage and all building blocks of the last stage.
- Ablation Studies: Replacing strip pooling with global average pooling lowers performance from 41.92% to 41.34%, below the baseline.The paper suggests direct feature fusion into a 1D vector loses spatial information and creates ambiguity.
5. Conclusions
The paper introduces strip pooling and builds strip- and spatial-pooling modules around it for scene parsing. Experiments on widely used datasets demonstrate the effectiveness of the approach.
- Conclusions: Strip pooling uses a long but narrow window to collect global contextual information relevant to scene parsing.The operation is presented as a new type of spatial pooling.
- Conclusions: The strip pooling module enlarges the backbone receptive field, while the mixed pooling module combines strip and spatial pooling in a residual bottleneck block.Both modules are introduced as pooling-based designs for scene parsing networks.
- Conclusions: Experiments on several widely used datasets demonstrate the effectiveness of the proposed approach.The paper reports comparisons including Pascal Context state-of-the-art results.