PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Atrous Convolution for Semantic Image Segmentation

Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam

arXiv:1706.05587v3cs.CV

TL;DR

Semantic segmentation needs dense spatial detail and multi-scale context, which standard convolutional networks can lose through downsampling. DeepLabv3 revisits atrous convolution with cascaded and parallel multi-rate modules plus image-level features, significantly improving over earlier DeepLab versions and matching other state-of-the-art models on PASCAL VOC 2012.

  • Problem

    Semantic segmentation requires detailed spatial information and multi-scale context, but repeated pooling and striding reduce feature-map resolution in deep convolutional networks.

  • Method

    DeepLabv3 uses atrous convolution in cascaded and spatial-pyramid modules with multiple rates, augmenting ASPP with image-level features to capture dense, multi-scale context.

  • Results

    DeepLabv3 significantly improves over previous DeepLab versions and achieves comparable performance with other state-of-the-art models on the PASCAL VOC 2012 semantic segmentation benchmark.

  • Takeaways & Limitations

    The results support atrous convolution with multi-rate cascaded and pyramid modules as an effective approach for dense semantic image segmentation.

  • Takeaways & Limitations

    At very large atrous rates, ASPP can degenerate to a 1 × 1 filter because image-boundary effects leave only the center weight effective.

Abstract

from arXiv · show

In this work, we revisit atrous convolution, a powerful tool to explicitly adjust filter's field-of-view as well as control the resolution of feature responses computed by Deep Convolutional Neural Networks, in the application of semantic image segmentation. To handle the problem of segmenting objects at multiple scales, we design modules which employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous rates. Furthermore, we propose to augment our previously proposed Atrous Spatial Pyramid Pooling module, which probes convolutional features at multiple scales, with image-level features encoding global context and further boost performance. We also elaborate on implementation details and share our experience on training our system. The proposed `DeepLabv3' system significantly improves over our previous DeepLab versions without DenseCRF post-processing and attains comparable performance with other state-of-art models on the PASCAL VOC 2012 semantic image segmentation benchmark.

1. Introduction

The introduction identifies reduced feature resolution and multiple object scales as central challenges for semantic segmentation. It revisits atrous convolution in cascaded and parallel modules, including ASPP augmented with image-level features to capture multi-scale context.

  • Challenges: Semantic segmentation must preserve detailed spatial information despite reduced feature resolution caused by pooling operations and convolution striding.The resulting invariance to local image transformation can impede dense prediction tasks.
  • Challenges: Objects appearing at multiple scales motivate approaches such as image pyramids, encoder-decoder structures, and multi-scale context architectures.The passage organizes existing methods into four categories, with the remaining categories continuing beyond the supplied excerpt.
  • Atrous convolution: Atrous convolution enlarges filter field-of-view and incorporates multi-scale context without requiring extra learned parameters.Standard convolution is the rate = 1 case, while larger atrous rates enable object encoding at multiple scales.
  • Proposed modules: The proposed modules combine atrous convolutions with various rates and batch normalization, arranged either in cascade or in parallel through Atrous Spatial Pyramid Pooling.The authors specifically experiment with both cascade layouts and the parallel ASPP method.
  • ASPP augmentation: Extremely large rates in 3 × 3 atrous convolutions can fail to capture long-range information because image boundary effects make them degenerate to 1 × 1 convolution.The proposed remedy is to incorporate image-level features into ASPP.

2. Related Work

Related work uses context to improve semantic segmentation through image pyramids, encoder-decoder designs, context modules, and spatial pyramid pooling. This work focuses on atrous convolution as both a context module and a tool for spatial pyramid pooling, revisiting cascaded and parallel designs.

  • Context-based FCNs: Four FCN designs exploit context information for semantic segmentation: image pyramids, encoder-decoders, context modules, and spatial pyramid pooling.Global features or contextual interactions have been shown beneficial for correctly classifying segmentation pixels.
  • Image pyramid: Image pyramids apply shared-weight models to multi-scale inputs, combining long-range context from small scales with small-object detail from large scales.Examples use Laplacian pyramids or sequential coarse-to-fine processing.
  • Encoder-decoder: Encoder-decoder models reduce feature-map resolution to capture longer-range information, then recover spatial dimensions and object details through a decoder.Examples include learned deconvolution and SegNet’s reuse of encoder pooling indices.
  • Context module: Context modules add cascaded components that encode long-range context, including DenseCRF-based methods, jointly trained CRF-DCNN systems, and extra convolutional layers.These approaches place additional processing on or above DCNN feature or belief maps.
  • Spatial pyramid pooling: Spatial pyramid pooling captures context at several ranges; DeepLabv2’s ASPP uses parallel atrous convolutions with different rates to capture multi-scale information.Image-level features in ParseNet provide global context, while PSP uses pooling at several grid scales.
  • Atrous convolution: This work explores atrous convolution as a context module and spatial-pyramid-pooling tool, using cascaded copies of ResNet’s last block and revisiting parallel ASPP.The framework is described as general because it could be applied to any network.

3. Methods

The methods use atrous convolution to preserve or control feature resolution while adapting the filter field-of-view, and develop cascaded and parallel modules for multi-scale context. The improved ASPP combines multiple atrous rates with image-level features to address reduced valid sampling at large rates and incorporate global context.

  • Repeated pooling and striding typically reduce DCNN feature-map resolution by a factor of 32 in each spatial direction.
  • Atrous convolution samples inputs with rate r, inserts r −1 zeros between filter values, and changes the filter’s field-of-view; standard convolution uses r = 1.
  • Atrous convolution explicitly controls feature-response density through the output stride, defined as input-image resolution divided by final output resolution.
  • Cascade modules: Cascaded replicas of ResNet block4 extend through block7, using striding for long-range information and Multi Grid unit rates across their three convolutional layers.Without atrous convolution, the cascade reaches output stride = 256.
  • Parallel modules: Improved ASPP uses one 1×1 and three 3×3 convolutions with rates = (6, 12, 18) at output stride = 16, plus an image-level feature branch.Larger atrous rates reduce valid filter weights, so global average-pooled features are projected with 256 filters and bilinearly upsampled.

4. Experimental Evaluation

The evaluation adapts ImageNet-pretrained ResNet with atrous convolution for PASCAL VOC 2012 segmentation and studies output stride, cascaded blocks, multi-grid modules, ASPP, and inference strategies. The best reported system reaches 86.9% on the PASCAL VOC 2012 test set, while achieving 79.77% on the validation set without DenseCRF post-processing or MS-COCO pre-training.

  • Implementation: The models use ImageNet-pretrained ResNet with atrous convolution, and output stride = 8 applies rates 2 and 4 in the final two blocks.The implementation is built on TensorFlow.
  • Cascaded Atrous Convolution: Atrous convolution improves ResNet-50 performance from 20.29% to 75.18% as output stride increases from 256 and corresponding atrous convolution is introduced.Output stride = 8 improves performance but requires more memory, whereas output stride = 16 is several times faster with coarser feature maps.
  • Cascaded Atrous Convolution: Adding cascaded blocks generally improves performance, but the gains diminish; block7 slightly hurts ResNet-50 while improving ResNet-101.Multi-grid is generally better than the vanilla configuration, simple rate doubling is ineffective, and deeper multi-grid improves performance.
  • ASPP: ASPP with Multi Grid = (1, 2, 4) improves over (1, 2, 1), while adding a rate = 24 branch reduces performance by 0.12%.The ASPP module is augmented with image-level features to encode global context.
  • Inference Strategy: 79.77% is achieved by the best ASPP model on the validation set, exceeding the best cascaded model at 79.35%.Output stride = 8 inference improves performance by 1.3% over output stride = 16; multi-scale inputs and left-right flips add 0.94% and 0.32%.

5. Conclusion

DeepLabv3 uses atrous convolution with upsampled filters to produce dense feature maps and capture long-range context. It encodes multi-scale information through cascaded atrous rates and an image-level-feature-augmented atrous spatial pyramid pooling module.

  • DeepLabv3 employs atrous convolution with upsampled filters to extract dense feature maps and capture long-range context.
  • The cascaded module encodes multi-scale information by gradually doubling the atrous rates.
  • The augmented atrous spatial pyramid pooling module probes features with filters at multiple sampling rates and effective field-of-views.

A. Effect of hyper-parameters

DeepLabv3 performance on PASCAL VOC 2012 depends strongly on the training protocol, especially batch-normalization fine-tuning, logit upsampling, and crop size. Larger batch sizes and output stride 16 during training further improve performance.

  • New training protocol: 77.21% performance is attained on the PASCAL VOC 2012 validation set with the new training protocol.The protocol uses larger crops, upsampled logits during training, and fine-tuned batch normalization.
  • New training protocol: 75.95% performance results without fine-tuning batch normalization, compared with 77.21% using the new protocol.The comparison is made on the PASCAL VOC 2012 validation set at output stride=16.
  • New training protocol: 76.01% performance results without upsampling logits during training, compared with 77.21% using the new protocol.In this setting, the ground truths are downsampled instead.
  • New training protocol: Smaller crop size, using 321 as in prior work, decreases performance relative to the new training protocol.The supplied passage states that performance decreases but does not provide the resulting value.
  • Varying batch size: Small batch sizes are inefficient for training DeepLabv3, whereas larger batch sizes lead to better performance with fine-tuned batch normalization.Table 9 uses output stride=16 during both training and evaluation.
  • Output stride: Using output stride=16 during training leads to better performance for evaluation output stride values of 8 and 16.Output stride controls feature-map resolution and the largest batch size that fits during training.

B. Asynchronous training

Asynchronous training with multiple replicas substantially reduces DeepLabv3 training time on PASCAL VOC 2012 semantic segmentation without degrading performance relative to the single-replica baseline.

  • B. Asynchronous training: 2.74 hours with 32 replicas reduces training time from the 3.65 days single-replica baseline without performance drop.The baseline uses one replica and a K80 GPU; experiments use TensorFlow asynchronous training on PASCAL VOC 2012.

C. DeepLabv3 on Cityscapes dataset

The section evaluates DeepLabv3 on Cityscapes using finely and coarsely annotated training sets, varying output stride and inference scales to improve validation performance. Cityscapes evaluation uses 19 semantic labels across 5,000 finely annotated images.

  • Dataset and evaluation: Cityscapes provides 5,000 finely annotated images and about 20,000 coarsely annotated images, with 19 semantic labels used for evaluation.The finely annotated images are split into 2,975 training, 500 validation, and 1,525 test images.
  • Train fine evaluation: 77.23% is achieved by DeepLabv3 on the validation set when trained with only the 2,975-image train fine set at output stride = 16.Training uses 90K iterations, crop size 769, and whole-image inference.
  • Trainval coarse evaluation: Training on the trainval coarse set combines 3,475 finely annotated images with 20,000 coarsely annotated images for comparison with state-of-the-art models.Inference uses scales = {0.75, 1, 1.25, 1.5, 1.75, 2} and evaluation output stride = 4 with CPUs.
  • Trainval coarse evaluation: 0.8% is contributed by using more inference scales, while 0.1% is contributed by evaluating at output stride = 4 on the validation set.These gains are reported relative to using only three scales and a coarser output-stride setting.
Loading 1706.05587v3…