Source-linked AI summary
Large Kernel Matters -- Improve Semantic Segmentation by Global Convolutional Network
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, Jian Sun
TL;DR
Semantic segmentation must reconcile pixel localization with transformation-robust classification, a tension that conventional designs do not fully address. The paper introduces GCN with large separable kernels and residual boundary refinement, reporting state-of-the-art results on PASCAL VOC 2012 and Cityscapes.
Problem
Semantic segmentation requires simultaneous per-pixel classification and localization, whose invariance and sensitivity requirements are naturally contradictory.
Method
GCN combines a fully convolutional architecture with large separable kernels, while Boundary Refinement models boundary alignment as a residual structure.
Results
82.2% on PASCAL VOC 2012 and 76.9% on Cityscapes are reported as state-of-the-art results.
Takeaways & Limitations
Large kernels improve the trade-off between valid receptive field and parameter count, while GCN mainly improves segmentation performance.
Takeaways & Limitations
The contribution of large kernels versus GCN structure remains uncertain, and the reason trivial large kernels become difficult to train still needs further study.
Abstract
from arXiv · showhide
One of recent trends [30, 31, 14] in network architec- ture design is stacking small filters (e.g., 1x1 or 3x3) in the entire network because the stacked small filters is more ef- ficient than a large kernel, given the same computational complexity. However, in the field of semantic segmenta- tion, where we need to perform dense per-pixel prediction, we find that the large kernel (and effective receptive field) plays an important role when we have to perform the clas- sification and localization tasks simultaneously. Following our design principle, we propose a Global Convolutional Network to address both the classification and localization issues for the semantic segmentation. We also suggest a residual-based boundary refinement to further refine the ob- ject boundaries. Our approach achieves state-of-art perfor- mance on two public benchmarks and significantly outper- forms previous results, 82.2% (vs 80.2%) on PASCAL VOC 2012 dataset and 76.9% (vs 71.8%) on Cityscapes dataset.
1. Introduction
Semantic segmentation must classify each pixel correctly while locating it precisely, but these goals impose conflicting transformation requirements. The paper proposes GCN and boundary refinement to address both simultaneously.
- Semantic segmentation requires per-pixel classification and coordinate-aligned localization in the output score map.
- Classification favors transformation invariance, whereas localization requires transformation sensitivity and precise pixel placement.
- GCN uses a fully convolutional architecture for localization and large kernels for densely connected per-pixel classification.
- Boundary refinement models boundary alignment as a residual structure integrated into the network and trained end-to-end.
- 82.2% on PASCAL VOC 2012 and 76.9% on Cityscapes are reported as state-of-the-art results.
2. Related Work
Prior semantic-segmentation research mainly advances context embedding, resolution enlargement, and boundary alignment. The paper instead frames segmentation as classification on large feature maps, using GCN to address classification and localization together.
- Semantic-segmentation research has pursued context embedding, resolution enlargement, and boundary alignment as major directions.
- Context Embedding: Context methods include hierarchical features, global pooling, multi-scale context, and atrous spatial pyramid pooling.
- Resolution Enlarging: Resolution methods increase prediction resolution through deconvolution, unpooling, learned upsampling, or dilated convolution.
- Boundary Alignment: Boundary-alignment methods commonly use CRF-based post-processing or end-to-end CRF-inspired operators, alongside bilateral alternatives.
- The paper treats semantic segmentation as classification on large feature maps and proposes GCN to fulfill classification and localization demands simultaneously.
3. Approach
The approach combines large-kernel global convolution with a fully convolutional segmentation framework to balance classification and localization. Boundary refinement further adjusts coarse score maps near object boundaries.
- The proposed GCN addresses the contradictory classification and localization requirements within a fully convolutional segmentation framework.
- Global Convolutional Network: Conventional segmentation classifiers may have insufficient valid receptive fields to cover enlarged objects, harming classification.
- Global Convolutional Network: GCN uses separable 1 × k and k × 1 convolutions to create dense connections across a large k×k region with practical computation and parameters.
- Overall Framework: The model combines pretrained ResNet features, FCN4, multi-scale score maps, and successive upsampling and addition.
- Global Convolutional Network: GCN significantly enlarges the valid receptive field compared with traditional segmentation models, including when the input object is resized.
- Overall Framework: The Boundary Refinement block outputs refined score map ˜S = S + R(S), modeling boundary alignment as a residual structure.
4. Experiment
Experiments evaluate GCN across benchmarks, kernel sizes, parameter controls, convolutional-stack alternatives, boundary regions, and pretrained models. GCN benefits from large kernels, improves internal-region accuracy, and achieves strong results on PASCAL VOC 2012 and Cityscapes.
- Experimental Setup: Experiments evaluate GCN on PASCAL VOC 2012 and Cityscapes using ablations followed by full benchmark results.The study uses standard mean IoU for evaluation and reports both validation and test-set outcomes.
- Global Convolutional Network — Large Kernel Matters: 5.5%: the k = 15 global-convolutional version surpasses the smallest tested kernel in standard mean IoU on PASCAL VOC 2012 validation.Performance consistently increases as kernel size grows from 3 to 15; k = 15 roughly matches the 16×16 top-most feature map.
- Global Convolutional Network — Large Kernel Matters: GCN outperforms trivial k × k convolutions despite using fewer parameters for each matched kernel size.The trivial convolution has more parameters, while GCN remains consistently better in performance.
- Global Convolutional Network — Large Kernel Matters: GCN outperforms equivalent stacks of 3 × 3 convolutions across large kernel sizes and retains better performance when the stack uses fewer parameters.The comparison removes intermediate nonlinearities, and reducing stack channels causes performance degradation.
- Boundary Refinement: GCN mainly improves internal-region accuracy, while Boundary Refinement mainly improves boundary-region accuracy.Boundary pixels are defined as those within distance ≤7 of object boundaries; the table reports per-pixel accuracy for boundary and internal regions.
- Global Convolutional Network for Pretrained Model: 5.5%: after segmentation fine-tuning, the GCN-based ResNet outperforms the original ResNet, although its ImageNet classification performance is slightly poorer.Adding GCN and boundary refinement makes the pretrained-model gain minor but still favorable.
- PASCAL VOC 2012: 82.2%: GCN + BR obtains this score on the PASCAL VOC 2012 test set and outperforms previous state-of-the-art results.Multi-scale post-processing and denseCRF also provide benefits in the reported experiments.
- Cityscapes: 76.9%: the best model obtains this score on the Cityscapes test set and surpasses previous publications.The result reaches a new state-of-the-art according to the authors.
5. Conclusion
The paper argues that large kernels help reconcile classification and localization in semantic segmentation, while Boundary Refinement improves object-boundary localization. The resulting GCN achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes.
- Large kernels help relieve the contradiction between classification and localization in semantic segmentation.
- The Global Convolutional Network balances valid receptive field and parameter count while improving segmentation performance.
- Boundary Refinement primarily improves performance near object boundaries, whereas GCN mainly improves internal regions.
- 82.2% on PASCAL VOC 2012 and 76.9% on Cityscapes are reported as state-of-the-art results.
Appendix.A ResNet50 and ResNet50-GCN
Table 11 describes the architectures of ResNet50 and ResNet50-GCN, including their component blocks, output sizes, downsampling, and convolutional complexity.
- Table 11 compares ResNet50 and ResNet50-GCN architectures using bottleneck and GCN blocks.
- Downsampling occurs between components through stride 2 convolution.
- Output sizes are measured using standard ImageNet 224 × 224 images.
- The table reports convolutional computational complexity in its last row.
Appendix.B Examples of semantic segmentation results on Cityscapes.
Figure 7 presents qualitative Cityscapes segmentation examples by placing the input image, GCN plus Boundary Refinement output, and ground truth side by side.
- Each row contains an input image, a GCN plus Boundary Refinement result, and ground truth.