Source-linked AI summary
FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation
Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, Yizhou Yu
TL;DR
Dilated convolutions provide high-resolution feature maps for semantic segmentation but impose heavy computation and memory costs. The paper introduces Joint Pyramid Upsampling to replace them, reporting over threefold complexity reduction without performance loss and strong benchmark results.
Problem
Dilated convolutions maintain high-resolution feature maps but bring heavy computation complexity and memory footprint.
Method
Joint Pyramid Upsampling formulates high-resolution feature extraction as joint upsampling and uses the last three backbone feature maps to generate a high-resolution feature map.
Results
The method reduces computation complexity by more than three times without performance loss and achieves state-of-the-art results across Pascal Context and ADE20K.
Takeaways & Limitations
JPU can replace dilated convolutions in several popular segmentation approaches while improving performance and running faster.
Abstract
from arXiv · showhide
Modern approaches for semantic segmentation usually employ dilated convolutions in the backbone to extract high-resolution feature maps, which brings heavy computation complexity and memory footprint. To replace the time and memory consuming dilated convolutions, we propose a novel joint upsampling module named Joint Pyramid Upsampling (JPU) by formulating the task of extracting high-resolution feature maps into a joint upsampling problem. With the proposed JPU, our method reduces the computation complexity by more than three times without performance loss. Experiments show that JPU is superior to other upsampling modules, which can be plugged into many existing approaches to reduce computation complexity and improve performance. By replacing dilated convolutions with the proposed JPU module, our method achieves the state-of-the-art performance in Pascal Context dataset (mIoU of 53.13%) and ADE20K dataset (final score of 0.5584) while running 3 times faster.
1. Introduction
Semantic segmentation networks seek high-resolution feature maps, but dilated convolutions impose substantial computation and memory costs. FastFCN replaces them with JPU while maintaining performance and improving speed.
- Semantic segmentation assigns a semantic label to every image pixel, commonly using Fully Convolutional Networks.
- The original FCN downsamples spatial resolution by a factor of 32, losing fine structure and causing inaccurate object-boundary predictions.
- EncoderDecoder methods recover spatial information with decoders, whereas DilatedFCNs preserve high-resolution feature maps using dilated convolutions.
- Dilated convolutions improve performance but substantially increase resource demands; some ResNet-101 residual blocks require 4 or 16 times more computation and memory.
- JPU replaces dilated convolutions, upsamples an OS=32 feature map to OS=8, and exploits multi-scale context across multi-level feature maps without performance loss.
- More than 3 times faster performance is reported, including mIoU of 53.13% on Pascal Context and state-of-the-art results on ADE20K.
2. Related Work
Related segmentation methods mainly follow DilatedFCN or EncoderDecoder designs, while JPU reframes high-resolution feature extraction as guided joint upsampling for high-dimensional feature maps.
- DilatedFCN: DilatedFCNs preserve receptive fields with dilated convolutions and process high-level features using multi-scale context modules.
- EncoderDecoder: EncoderDecoders combine multi-level encoder features with decoder activations to gradually recover spatial information.
- DilatedFCN: JPU replaces DilatedFCN backbone dilated convolutions and reduces computation complexity without performance loss.
- Joint Upsampling: JPU performs joint upsampling by using high-resolution feature maps as guidance for a low-resolution feature map.
- Joint Upsampling: Unlike prior joint-upsampling methods designed for 3-channel images, JPU processes feature maps with many channels and complex high-dimensional relationships.
- Data-Dependent Upsampling: DUpsampling depends strongly on the segmentation label space and generalizes poorly to larger or more complex label spaces.
3. Method
The method replaces backbone dilated convolutions with JPU, reformulating high-resolution feature extraction as joint upsampling while preserving semantic segmentation performance and reducing resource use.
- 3.2. The Framework of Our Method: DilatedFCNs preserve high-resolution feature maps and receptive fields, but their enlarged feature maps impose heavy computation and memory costs.The method targets this overload while seeking performance comparable to DilatedFCNs.
- 3.2. The Framework of Our Method: The backbone restores stride convolutions and replaces dilated convolutions with regular convolutions, reducing the five feature-map resolutions progressively as in the original FCN.JPU then processes Conv3–Conv5 to produce a feature map similar to DilatedFCN’s final feature map.
- 3.2. The Framework of Our Method: Compared to DilatedFCN, the method uses 4 times fewer computation and memory resources in 23 residual blocks and 16 times fewer in 3 blocks of ResNet-101.The authors report that this allows faster execution with lower memory consumption.
- 3.3.1 Background: Joint upsampling transfers structural details from high-resolution guidance features to generate a high-resolution target from low-resolution features.The approximation uses a lower-complexity transformation learned from low-resolution guidance and target features.
- 3.3.3 Solving with CNNs: Each JPU input is projected into a shared reduced-dimensional embedding, upsampled, concatenated, and processed by separable convolutions with dilation rates 1, 2, 4, and 8.The parallel branches extract multi-scale context from the fused multi-level features.
- 3.3.3 Solving with CNNs: JPU jointly upsamples Conv4 using Conv3 and Conv5 using the enlarged Conv4, solving two related joint upsampling problems.This design uses multiple feature stages rather than only the last feature map.
4. Experiment
The experiments evaluate JPU through ablations, efficiency measurements, generalization tests, and comparisons on Pascal Context and ADE20K. JPU improves segmentation performance while substantially reducing computation time and memory demands.
- Ablation Study: The experiments evaluate JPU against bilinear upsampling, FPN, and dilated-convolution baselines using segmentation accuracy and efficiency metrics.The ablation study reports pixel accuracy and mIoU, while FPS measures computation complexity.
- Ablation Study: JPU outperforms both FPN and EncNet by more than 1% in mIoU, while visual results show more accurate labeling of branches and side shoots.The visual comparison attributes the side-shoot result to extracting multi-scale context from multi-level feature maps.
- Ablation Study: JPU generalizes to DeepLabV3 and PSPNet, with transformed versions consistently outperforming their corresponding original methods.The replacements target popular DilatedFCN approaches.
- Ablation Study: For ResNet-50, JPU runs about two times faster than EncNet; for ResNet-101, it runs more than three times faster while achieving better performance than FPN.FPS is measured on a Titan-Xp GPU using 512 × 512 inputs and averaged over 100 runs.
- Comparison with Other Methods: On Pascal Context, JPU achieves state-of-the-art performance, including 53.13% mIoU, while ADE20K results include 42.75% mIoU with ResNet-50 and a 0.5584 final score with ResNet-101.The Pascal Context comparison also reports a 1.9% mIoU improvement when replacing ResNet-50 with ResNet-101.
5. Conclusion
The paper proposes JPU to replace dilated convolutions, reducing computation complexity by more than three times without performance loss while improving speed and segmentation performance.
- JPU formulates high-resolution feature extraction as a joint upsampling problem and replaces time- and memory-consuming dilated convolutions.
- More than three times lower computation complexity is achieved without performance loss using JPU.
- Ablation studies show that JPU is superior to other upsampling modules.
- Plugging JPU into modern segmentation approaches improves performance while making them run much faster.
- Experiments on two segmentation datasets report state-of-the-art performance with dramatically reduced computation complexity.
More Visual Results
Figures 8–12 present additional visual results of the ResNet-101 method on the Pascal Context and ADE20K validation sets.
- Figures 8–12 provide additional visual results of the method.
- Figures 8–10 show visual results on the Pascal Context validation set using ResNet-101.
- Figures 11–12 show visual results on the ADE20K validation set using ResNet-101.