Source-linked AI summary
Understanding Convolution for Semantic Segmentation
Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, Garrison Cottrell
TL;DR
Pixel-wise semantic segmentation needs accurate dense predictions despite downsampled feature maps and sparse dilated convolutions. The paper introduces DUC for learnable decoding and HDC for mitigating gridding while enlarging receptive fields. The combined framework achieves state-of-the-art results across Cityscapes, KITTI, and PASCAL VOC2012, including 80.1% mIoU on Cityscapes at submission.
Problem
Pixel-wise semantic segmentation requires recovering detailed H × W label maps from downsampled representations, while bilinear upsampling is not learnable and may lose fine details.
Method
The paper combines learnable Dense Upsampling Convolution for dense decoding with Hybrid Dilated Convolution for enlarging receptive fields and alleviating gridding.
Results
80.1% mIoU was achieved on the Cityscapes test set with a single model without CRF post-processing, reported as state-of-the-art at submission.
Takeaways & Limitations
The framework’s practical consequence is improved pixel-level semantic segmentation across Cityscapes, KITTI road estimation, and PASCAL VOC2012 tasks.
Abstract
from arXiv · showhide
Recent advances in deep learning, especially deep convolutional neural networks (CNNs), have led to significant improvement over previous semantic segmentation systems. Here we show how to improve pixel-wise semantic segmentation by manipulating convolution-related operations that are of both theoretical and practical value. First, we design dense upsampling convolution (DUC) to generate pixel-level prediction, which is able to capture and decode more detailed information that is generally missing in bilinear upsampling. Second, we propose a hybrid dilated convolution (HDC) framework in the encoding phase. This framework 1) effectively enlarges the receptive fields (RF) of the network to aggregate global information; 2) alleviates what we call the "gridding issue" caused by the standard dilated convolution operation. We evaluate our approaches thoroughly on the Cityscapes dataset, and achieve a state-of-art result of 80.1% mIOU in the test set at the time of submission. We also have achieved state-of-the-art overall on the KITTI road estimation benchmark and the PASCAL VOC2012 segmentation task. Our source code can be found at https://github.com/TuSimple/TuSimple-DUC .
1. Introduction
The paper improves pixel-wise semantic segmentation by redesigning convolutional operations for decoding and encoding. DUC replaces non-learnable bilinear upsampling, while HDC addresses gridding in dilated convolution and supports strong results across benchmarks.
- Decoding: DUC replaces bilinear upsampling with learnable dense prediction filters that recover finer pixel-level information, especially for small objects.Bilinear upsampling may lose fine details, whereas DUC learns to upscale downsized feature maps into the desired dense feature map.
- Encoding: Dilated convolution preserves feature-map resolution and receptive field, but its sparse sampling can create checkerboard-like gridding and lose neighboring information.The problem becomes worse as dilation rates increase in higher layers.
- Contribution: The proposed framework combines DUC and HDC to improve convolution operations for pixel-level semantic segmentation.The authors evaluate the combined approach with CRF post-processing across Cityscapes, KITTI, and PASCAL VOC2012.
2. Related Work
Prior semantic segmentation systems decode downsampled features with interpolation or deconvolution and use dilated convolution to preserve resolution and aggregate context.
- Decoding of Feature Representation: Bilinear interpolation is commonly used because it is fast and memory-efficient, while deconvolution methods recover predictions through learned or unpooling-based decoding.Earlier systems also used single or multiple deconvolutional layers for prediction or generation tasks.
- Dilated Convolution: Dilated convolution inserts holes into kernels to increase resolution and enable dense feature extraction in deep CNNs.Prior work used increasing dilation rates for context aggregation and atrous spatial pyramid structures.
3. Our Approach
The approach introduces DUC for learnable dense decoding and HDC for broader, less sparse receptive fields during encoding. Together, these operations produce pixel-level predictions while addressing information loss from interpolation and gridding from repeated dilation.
- 3.1. Dense Upsampling Convolution (DUC): DUC applies convolution directly to a downsampled h × w × c feature map and reshapes h × w × (d^2 × L) outputs into an H × W × L label map.The operation divides the full label map into d^2 subparts and predicts each pixel through learned convolutional outputs.
- 3.1. Dense Upsampling Convolution (DUC): Because DUC is learnable, it can recover fine details that bilinear interpolation often misses, including objects smaller than the network’s downsampling stride.The paper gives a 1/16 downsampling rate as an example where poles or distant people may not be recovered by bilinear upsampling.
- 3.2. Hybrid Dilated Convolution (HDC): Dilated convolution maintains feature-map resolution and receptive field by inserting zeros between kernel elements instead of using downsampling.For a k × k kernel, the resulting dilated filter has size k_d × k_d, where k_d = k + (k − 1) · (r − 1).
- 3.2. Hybrid Dilated Convolution (HDC): With k = 3 and r = 2, only 9 of 25 positions contribute, producing checkerboard sampling and inconsistent local information across nearby regions.Repeated equal dilation rates restrict the top-layer receptive field to a sparse subset of locations.
- 3.2. Hybrid Dilated Convolution (HDC): HDC selects varying dilation rates so successive receptive fields cover a square region without holes or missing edges.Its design uses a maximum-distance criterion and a sawtooth-like rate pattern, such as 1, 2, and 3 across successive layers.
- 3.2. Hybrid Dilated Convolution (HDC): HDC enlarges receptive fields without extra modules, but rates within a group should not share common factors such as 2, 4, and 8.The method is integrated into the original network layers and is intended to help recognize relatively large objects.
4. Experiments and Results
Experiments evaluate DUC and HDC through Cityscapes ablations and test results, alongside KITTI and PASCAL VOC2012 benchmarks. The proposed models improve segmentation performance, with gains from dense upsampling, larger patches, coarse labels, and architectural changes.
- Dense Upsampling Convolution (DUC): 74.3% mIoU on Cityscapes validation improves the 72.3% baseline by 2% after replacing the top convolution with DUC.DUC adds parameters only at the top convolutional layer and reshapes its output into the final prediction map.
- Dense Upsampling Convolution (DUC): DUC particularly improves identification of small objects such as poles, traffic lights, and traffic signs compared with bilinear interpolation.The authors attribute this to pixel-level dense upsampling recovering detailed information missed by bilinear interpolation.
- Ablation Studies: 75.7% validation performance results from training with an 880 × 880 patch size, a 1% improvement over the previous best.The larger patches are intended to aggregate local detail and global context while reducing GPU memory through cell = 2.
- Hybrid Dilated Convolution (HDC): Increasing receptive-field size generally yields higher accuracy, while HDC is reported to eliminate the gridding effect and perform particularly well on relatively large objects.The HDC variants were applied starting from the best 101-layer ResNet-DUC model.
- Test Set Results: 78.5% mIoU is achieved on the Cityscapes test set with coarse data, compared with 77.6% mIoU using fine data only.These results use the ResNet-DUC-HDC model.
- Test Set Results: 80.1% mIoU is achieved on Cityscapes test data using a single model without CRF post-processing, while KITTI and PASCAL VOC2012 results are reported as state of the art at submission.The PASCAL VOC2012 model reaches 83.1% mIoU using a single model without ensemble or multiscale testing.
5. Conclusion
The paper introduces DUC for pixel-level prediction and HDC for addressing gridding while enlarging receptive fields in semantic segmentation.
- DUC enables pixel-level prediction on feature maps for semantic segmentation.
- HDC addresses the gridding problem caused by standard dilated convolution while effectively enlarging the network’s receptive fields.
- The proposed convolutional operations achieve effective improvements across various semantic segmentation tasks.