Source-linked AI summary
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam
TL;DR
Semantic segmentation models must balance multi-scale contextual encoding with recovery of detailed object boundaries. DeepLabv3+ combines these strengths with an encoder-decoder design and reports 89.0% on PASCAL VOC 2012 and 82.1% on Cityscapes without post-processing.
Problem
Semantic segmentation models encode rich context but lose detailed object-boundary information through pooling and strided convolutions.
Method
DeepLabv3+ combines DeepLabv3’s atrous multi-scale encoder with a decoder for boundary recovery and atrous separable convolutions using Xception.
Results
89.0% on PASCAL VOC 2012 and 82.1% on Cityscapes were achieved without post-processing, setting new state-of-the-art performance.
Takeaways & Limitations
The proposed encoder-decoder network achieves state-of-the-art semantic segmentation performance on PASCAL VOC 2012 and Cityscapes.
Takeaways & Limitations
The model has difficulty distinguishing sofas from chairs, segmenting heavily occluded objects, and handling objects seen from rare viewpoints.
Abstract
from arXiv · showhide
Spatial pyramid pooling module or encode-decoder structure are used in deep neural networks for semantic segmentation task. The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information. In this work, we propose to combine the advantages from both methods. Specifically, our proposed model, DeepLabv3+, extends DeepLabv3 by adding a simple yet effective decoder module to refine the segmentation results especially along object boundaries. We further explore the Xception model and apply the depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules, resulting in a faster and stronger encoder-decoder network. We demonstrate the effectiveness of the proposed model on PASCAL VOC 2012 and Cityscapes datasets, achieving the test set performance of 89.0\% and 82.1\% without any post-processing. Our paper is accompanied with a publicly available reference implementation of the proposed models in Tensorflow at \url{https://github.com/tensorflow/models/tree/master/research/deeplab}.
1 Introduction
DeepLabv3+ combines spatial pyramid pooling with an encoder-decoder structure to preserve rich semantic context while recovering detailed object boundaries. It further applies atrous separable convolution and an adapted Xception model, achieving 89.0% on PASCAL VOC 2012 and 82.1% on Cityscapes without post-processing.
- Model motivation: DeepLabv3+ extends DeepLabv3 with a simple decoder that recovers object boundaries from rich semantic encoder features.Atrous convolution controls encoder-feature density according to computational resources.
- Architectural extensions: The model adapts Xception for segmentation and applies depthwise separable convolution to both ASPP and decoder modules.The resulting encoder-decoder network improves speed and accuracy.
- Results: 89.0% is achieved on PASCAL VOC 2012 and 82.1% on Cityscapes without any post-processing.The paper reports these as test-set performances and describes them as state-of-the-art.
- Model motivation: Atrous convolution enables arbitrary control over extracted-feature resolution, trading precision against runtime.This flexibility is described as unavailable in existing encoder-decoder models.
- Availability: The authors publicly release a TensorFlow implementation of the proposed models.The implementation is provided through the DeepLab repository.
2 Related Work
Related segmentation work mainly exploits multi-scale contextual information through spatial pyramid pooling or atrous convolutions, while encoder-decoder networks recover spatial detail through separate encoding and decoding stages. Depthwise separable convolution reduces computation and parameters while maintaining similar or slightly better performance, motivating its exploration in Xception-based designs.
- Spatial pyramid pooling: Spatial pyramid models use multi-scale pooling or parallel atrous convolutions to exploit contextual information for semantic segmentation.Examples include PSPNet and DeepLab, including image-level pooling and Atrous Spatial Pyramid Pooling.
- Encoder-decoder: Encoder-decoder networks reduce feature-map resolution to capture semantics, then progressively recover spatial information.This structure has been applied to tasks including human pose estimation, object detection, and semantic segmentation.
- Encoder-decoder: DeepLabv3+ combines multi-scale atrous encoding with a decoder that refines segmentation results along object boundaries.The encoder applies atrous convolution at multiple scales, while the decoder progressively improves boundary detail.
- Depthwise separable convolution: Depthwise separable convolution reduces computation cost and parameter count while maintaining similar or slightly better performance.The operation has been adopted in multiple recent neural-network designs and is explored here with the Xception model.
3 Methods
The method combines atrous convolution and depthwise separable convolution in a DeepLabv3 encoder-decoder architecture. It uses a decoder to recover segmentation details and a modified Xception model to improve performance with faster computation.
- Modified Xception model: A modified Xception model adapts prior Xception modifications for semantic segmentation, further improving performance with faster computation.The method builds on MSRA’s Aligned Xception modifications and introduces additional changes for semantic image segmentation.
- Atrous convolution: Atrous convolution controls feature-map resolution and filter field-of-view through the atrous rate r, enabling multi-scale information capture.Standard convolution is the special case r = 1, while changing r adaptively modifies the filter’s field-of-view.
- Depthwise separable convolution: Depthwise separable convolution factorizes standard convolution into channel-wise depthwise and 1 × 1 pointwise convolutions, drastically reducing computation complexity.The depthwise convolution applies spatial filters independently per input channel, and the pointwise convolution combines their outputs.
- DeepLabv3 as encoder: DeepLabv3 serves as the encoder, using atrous convolution and Atrous Spatial Pyramid Pooling to extract features at arbitrary resolution and multiple scales.The proposed structure uses the last feature map before logits as its encoder output; this feature map has 256 channels and rich semantic information.
- Proposed decoder: The proposed decoder first upsamples encoder features by 4 and concatenates them with earlier features to recover object-segmentation details more effectively than naive 16-fold upsampling.DeepLabv3’s naive decoder bilinearly upsamples features by a factor of 16, but may not successfully recover segmentation details.
4 Experimental Evaluation
Experiments show that DeepLabv3+ improves segmentation through decoder design, while depthwise separable convolutions substantially reduce computation with similar accuracy. The model achieves 89.0% on PASCAL VOC 2012 and 82.1% on Cityscapes test sets without post-processing.
- Inference evaluation: 77.21% is the ResNet-101 DeepLabv3 baseline, rising to 78.85% or 79.35% after adding the decoder at evaluation output strides 16 or 8, respectively.The decoder adds about 20B computation, with further gains from multi-scale and flipped inputs.
- Decoder design: Reducing low-level feature channels to 48 with a 1 × 1 convolution improves performance, motivating the adopted [1 × 1, 48] decoder design.The comparison uses Conv2 features and fixes the remaining decoder components to [3 × 3, 256].
- Decoder design: Two 3 × 3 convolutions with 256 filters outperform one or three convolutions in the decoder, while 128 filters or 1 × 1 kernels degrade performance.The preferred structure uses Conv2 features before striding and two extra [3 × 3, 256] operations.
- Efficiency: 33% to 41% lower Multiply-Adds are achieved by using depthwise separable convolutions in ASPP and decoder modules, while obtaining similar mIOU.The comparison evaluates DeepLabv3+ variants with ResNet-101 and reports both mIOU and Multiply-Adds.
- Benchmark results: 89.0% is achieved on the PASCAL VOC 2012 test set with JFT pretraining, while Cityscapes performance reaches 82.1% on the test set.On VOC, the corresponding result without JFT pretraining is 87.8%; the Cityscapes result is described as state of the art.
5 Conclusion
DeepLabv3+ combines DeepLabv3’s contextual encoder with a decoder that recovers object boundaries. The model also supports arbitrary-resolution atrous feature extraction and uses Xception with atrous separable convolution for greater speed and strength.
- 5 Conclusion: DeepLabv3+ combines DeepLabv3 as a contextual encoder with a decoder module that recovers object boundaries.The decoder is described as simple yet effective.
- 5 Conclusion: Atrous convolution can extract encoder features at arbitrary resolutions according to available computation resources.
- 5 Conclusion: Xception and atrous separable convolution make DeepLabv3+ faster and stronger.