Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation

Zhi Tian; Tong He; Chunhua Shen; Youliang Yan

Source-linked AI summary

Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation

Zhi Tian, Tong He, Chunhua Shen, Youliang Yan

arXiv:1903.02120v3Mar 5, 2019cs.CV

TL;DR

Semantic-segmentation decoders commonly rely on data-independent bilinear upsampling, which limits accurate recovery from coarse representations and constrains feature aggregation. The paper replaces it with DUpsampling, achieving efficient decoders and state-of-the-art results on PASCAL VOC and PASCAL Context.

Problem
Bilinear upsampling is data independent and has limited capability to recover pixel-wise predictions accurately, forcing higher-resolution decoder outputs and constraining feature aggregation.
Method
The paper proposes DUpsampling, which uses redundancy in the segmentation label space to reconstruct full-resolution predictions from coarse CNN outputs and enables low-resolution feature fusion.
Results
∼20% decoder computation, 88.1% mIOU on PASCAL VOC with 30% of DeepLabv3+ computation, and 52.5% mIOU on PASCAL Context.
Takeaways & Limitations
DUpsampling produces a flexible, light-weight decoder that combines effective segmentation with substantially reduced computation and memory demands.
Takeaways & Limitations
Vanilla softmax combined with DUpsampling can produce insufficiently sharp activation, causing cross-entropy loss to stall and training to converge slowly.

Abstract

from arXiv · show

Recent semantic segmentation methods exploit encoder-decoder architectures to produce the desired pixel-wise segmentation prediction. The last layer of the decoders is typically a bilinear upsampling procedure to recover the final pixel-wise prediction. We empirically show that this oversimple and data-independent bilinear upsampling may lead to sub-optimal results. In this work, we propose a data-dependent upsampling (DUpsampling) to replace bilinear, which takes advantages of the redundancy in the label space of semantic segmentation and is able to recover the pixel-wise prediction from low-resolution outputs of CNNs. The main advantage of the new upsampling layer lies in that with a relatively lower-resolution feature map such as $\frac{1}{16}$ or $\frac{1}{32}$ of the input size, we can achieve even better segmentation accuracy, significantly reducing computation complexity. This is made possible by 1) the new upsampling layer's much improved reconstruction capability; and more importantly 2) the DUpsampling based decoder's flexibility in leveraging almost arbitrary combinations of the CNN encoders' features. Experiments demonstrate that our proposed decoder outperforms the state-of-the-art decoder, with only $\sim$20\% of computation. Finally, without any post-processing, the framework equipped with our proposed decoder achieves new state-of-the-art performance on two datasets: 88.1\% mIOU on PASCAL VOC with 30\% computation of the previously best model; and 52.5\% mIOU on PASCAL Context.

1. Introduction

The paper identifies bilinear upsampling as a data-independent bottleneck that constrains feature aggregation and increases computation. It proposes DUpsampling to recover predictions from coarse outputs, enabling flexible, efficient decoders with strong segmentation results.

Motivation: Bilinear upsampling cannot accurately recover pixel-wise predictions because it ignores correlations among pixels and is data independent.This forces convolutional decoders to produce relatively high-resolution feature maps.
Motivation: The bilinear constraint forces encoder strides to be reduced aggressively, increasing computation and memory demands.DeepLabv3+ reduced encoder strides from 32 to 8 to achieve state-of-the-art performance, making inference slow.
DUpsampling: DUpsampling replaces bilinear upsampling and reconstructs pixel-wise predictions from relatively coarse CNN outputs using redundancy in the segmentation label space.This alleviates the need for precise high-resolution responses from the convolutional decoder.
Flexible decoder: DUpsampling decouples fused-feature resolution from final-prediction resolution, enlarging the design space for feature aggregation.Fused features can be downsampled to the lowest feature-map resolution before merging.
Results: The proposed decoder uses ∼20% of state-of-the-art decoder computation while achieving 88.1% mIOU on PASCAL VOC and 52.5% mIOU on PASCAL Context.On PASCAL VOC, it uses 30% of the computation of the previous best DeepLabv3+ framework.

2. Related Work

Prior semantic-segmentation approaches improve dense predictions mainly through atrous convolutions or encoder-decoder architectures. These approaches trade off receptive-field preservation, resolution, computation, memory, or decoder trainability.

Prior approaches: Semantic-segmentation methods broadly use atrous convolution or encoder-decoder architectures to improve pixel-wise predictions.These are presented as the two main groups of prior approaches.
Atrous convolution: Reducing backbone strides directly diminishes receptive fields, while atrous convolutions preserve receptive fields but require more computation and memory.Atrous convolutions avoid excessive feature-map downsampling while maintaining large receptive fields.
Encoder-decoder architectures: DeconvNet gradually recovers full-resolution predictions but is difficult to train because its decoder introduces many parameters.SegNet follows a similar recovery strategy using pooling indices.

3. Our Approach

The approach replaces bilinear upsampling with data-dependent DUpsampling, enabling accurate recovery from coarse CNN outputs and more flexible, efficient feature aggregation. It also uses adaptive-temperature softmax and downsampled low-level features to address optimization and decoder-resolution constraints.

3. Our Approach: Bilinear upsampling limits reconstruction quality and forces decoders to produce higher-resolution feature maps, increasing computation and restricting feature aggregation.Reducing overall strides from 16 to 8 incurs more than 3 times computation, while bilinear-based fusion narrows the available design space.
3. Our Approach: 16 or 32 is the typical spatial upsampling ratio because final CNN outputs are commonly 1/16 or 1/32 the ground-truth resolution.The low-resolution output must be restored to the label-map size before pixel-wise prediction or loss computation.
3. Our Approach: The method compresses local one-hot label vectors with a linear projection and reconstructs them using an inverse matrix learned by minimizing training-set reconstruction error.P compresses v into x, while W reconstructs x back to v; PCA provides a closed-form solution under an orthogonality constraint.
3. Our Approach: DUpsampling replaces bilinear upsampling to recover pixel-wise predictions from relatively coarse CNN outputs.It exploits redundancy in the segmentation label space and can be implemented through learned linear reconstruction.
3. Our Approach: Adaptive-temperature softmax learns its temperature through back-propagation, making DUpsampling training converge faster without extra hyper-parameter tuning.Vanilla softmax with DUpsampling can produce insufficiently sharp activations and become stuck during training.
3. Our Approach: Downsampling low-level features before fusion keeps decoder computation at the lowest resolution and decouples fused-feature resolution from final-prediction resolution.This permits arbitrary feature levels to be fused, but the scheme works only when combined with DUpsampling.

4. Experiments

Experiments show that DUpsampling improves reconstruction and enables flexible feature aggregation while reducing decoder and encoder computation. Across ablations and benchmarks, the proposed framework matches or exceeds bilinear-based alternatives with substantially less computation.

DUpsampling vs. Bilinear: DUpsampling’s upper-bound performance is well above bilinear upsampling at output strides 32 and 16.The upper bound measures recovery of pixel-wise predictions from CNN feature maps.
DUpsampling vs. Bilinear: 1.3 points and 1 point mIOU improvements are obtained on PASCAL VOC val when replacing bilinear at output strides 32 and 16, respectively.At output stride 32, DUpsampling achieves similar performance to bilinear at output stride 16.
Flexible aggregation of convolutional features: 74.20% mIOU is achieved by the ResNet-50 fusion combination conv1 3 + b3u6u3 on the PASCAL VOC val set.Computation changes little across feature levels, allowing fusion selection without considering resolution-related computation cost.
Comparison with the vanilla bilinear decoder: 74.03% mIOU with DUpsampling exceeds the vanilla bilinear decoder’s 73.26% using the same low-level features and ResNet-50 backbone.The reported improvement is attributed to DUpsampling rather than different low-level features.
Comparison with the vanilla bilinear decoder: 79.09% versus 78.70% is achieved with Xception-65 and the same low-level features, while better downsampled feature fusion benefits the proposed decoder more than vanilla bilinear.The vanilla decoder’s high-resolution feature fusion can run out of GPU memory.
Comparison with state-of-the-art Methods: 30% computation of DeepLabv3+ is required to achieve state-of-the-art PASCAL VOC performance, while the framework also sets a new state-of-the-art on PASCAL Context.The PASCAL VOC comparison uses output stride 16 instead of DeepLabv3+’s output stride 8.

5. Conclusion

The paper concludes that DUpsampling supports a flexible, lightweight decoder for semantic segmentation. By decoupling fused-feature and prediction resolutions, it expands feature-aggregation choices while reducing computation.

5. Conclusion: DUpsampling eliminates the need for computationally inefficient high-resolution feature maps from the underlying CNNs.The decoder produces pixel-wise predictions from coarse convolutional outputs.
5. Conclusion: Decoupling fused-feature resolution from final-prediction resolution allows almost arbitrary feature aggregation to improve segmentation performance.The decoder also avoids upsampling low-resolution high-level features to high-resolution low-level feature maps.
5. Conclusion: The proposed decoder combines effectiveness and efficiency, attaining state-of-the-art performance with less computation than previous state-of-the-art methods.

6. PASCAL VOC without COCO Pre-training

Without COCO pre-training, the proposed framework is evaluated on the PASCAL VOC test set using the stated training protocol and no post-processing.

6. PASCAL VOC without COCO Pre-training: The model is trained on SBD, fine-tuned on the official PASCAL VOC trainval set, and evaluated with multi-scale testing and left-right flipping.
6. PASCAL VOC without COCO Pre-training: No post-processing is used, and final performance is obtained by uploading test results to the official test server.
6. PASCAL VOC without COCO Pre-training: Table 7 reports state-of-the-art methods on the PASCAL VOC test set without COCO pre-training.

7. Visualization

Visualizations show reliable segmentation in challenging cases and fine-grained predictions despite low-resolution CNN feature maps and output stride 16.

7. Visualization: Small, distant, and incomplete objects can be segmented well without post-processing.
7. Visualization: Fine-grained segmentation is produced with low-resolution CNN output feature maps.The visualization uses output stride 16 during evaluation.