PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Context Aggregation by Dilated Convolutions

Fisher Yu, Vladlen Koltun

arXiv:1511.07122v3cs.CV

TL;DR

Semantic segmentation networks adapted from image classification may retain components unsuited to dense prediction. This paper introduces dilated-convolution context aggregation and simplifies adapted networks, achieving 67.6% versus 66.4% leaderboard accuracy against DeepLab+CRF without a CRF.

  • Problem

    Image-classification adaptations for dense prediction raise questions about which components are necessary and whether dedicated architectures can better support semantic segmentation.

  • Method

    The paper combines a plug-in dilated-convolution module for multi-scale context without resolution loss with a simplified prediction front end.

  • Results

    67.6% vs. 66.4% leaderboard accuracy: the simplified front end outperformed DeepLab+CRF without a CRF, while the context module reliably increased existing-system accuracy.

  • Takeaways & Limitations

    Dilated convolutions and removing image-classification vestiges can improve dense prediction architectures for semantic segmentation.

  • Takeaways & Limitations

    The authors note that state-of-the-art semantic segmentation systems still leave significant room for future advances and show failure cases of their most accurate configuration.

Abstract

from arXiv · show

State-of-the-art models for semantic segmentation are based on adaptations of convolutional networks that had originally been designed for image classification. However, dense prediction and image classification are structurally different. In this work, we develop a new convolutional network module that is specifically designed for dense prediction. The presented module uses dilated convolutions to systematically aggregate multi-scale contextual information without losing resolution. The architecture is based on the fact that dilated convolutions support exponential expansion of the receptive field without loss of resolution or coverage. We show that the presented context module increases the accuracy of state-of-the-art semantic segmentation systems. In addition, we examine the adaptation of image classification networks to dense prediction and show that simplifying the adapted network can increase accuracy.

1 INTRODUCTION

Semantic segmentation requires pixel-level accuracy together with multi-scale contextual reasoning, creating a mismatch with classification networks that reduce resolution through pooling and subsampling. This work introduces a dilated-convolution context module for dense prediction and evaluates it alongside simplified classification-network adaptations.

  • Semantic segmentation assigns a discrete or continuous label to every image pixel and must combine pixel-level accuracy with multi-scale contextual reasoning.
  • Repurposed image-classification convolutional networks substantially outperform prior semantic-segmentation methods.
  • Classification networks obtain multi-scale context through pooling and subsampling, whereas dense prediction requires multi-scale reasoning with full-resolution output.
  • The proposed context module aggregates multi-scale information without losing resolution or analyzing rescaled images, using a pooling-free, subsampling-free architecture based on dilated convolutions.
  • The work also re-examines leading image-classification adaptations in a controlled setting and removes vestigial components that can obscure core prediction-module performance.
  • Controlled Pascal VOC 2012 experiments show that plugging the context module into existing semantic-segmentation architectures reliably increases accuracy.

2 DILATED CONVOLUTIONS

This section defines dilated convolution as a generalized convolution controlled by a dilation factor and distinguishes it from ordinary convolution and the algorithme à trous. It motivates systematic dilation for multi-scale context aggregation through exponentially expanding receptive fields without losing resolution or coverage.

  • Definition: Dilated convolution, denoted ∗l, generalizes the discrete convolution operator using a dilation factor l.The passages introduce ∗l as the generalized operator and call it an l-dilated convolution.
  • Definition: Ordinary discrete convolution is the special case of 1-dilated convolution.The section explicitly identifies ∗ as 1-dilated convolution.
  • Definition: The operator modifies how filter parameters are used rather than constructing or representing a dilated filter.This terminology distinguishes dilated convolution from the phrase “convolution with a dilated filter.”
  • Receptive-field expansion: Systematic dilation uses exponentially increasing dilation factors to expand receptive fields exponentially without losing resolution or coverage.The architecture applies 3×3 filters with dilation 2^i, following Fi+1 = Fi ∗2i ki.
  • Receptive-field expansion: The receptive field after applying dilation 2^i has size (2^(i+2) −1)×(2^(i+2) −1).The section describes the receptive field as a square of exponentially increasing size and illustrates this progression in Figure 1.

3 MULTI-SCALE CONTEXT AGGREGATION

The context module aggregates multi-scale contextual information through dilated convolutions while preserving feature-map resolution, making it plug-compatible with dense prediction architectures. Even the basic module improves dense prediction accuracy with approximately 64C^2 parameters.

  • Module design: The module maps C input feature maps to C output feature maps in the same form, enabling direct integration into existing dense prediction architectures.Its purpose is to aggregate multi-scale contextual information for dense prediction.
  • Module design: The basic context module uses seven 3×3 convolutional layers with dilation factors 1, 1, 2, 4, 8, 16, and 1, followed by a 1×1 output layer.Each dilated convolution operates across all C feature maps and is followed by pointwise truncation max(·, 0).
  • Initialization: Identity initialization sets each layer to pass inputs directly forward, addressing the failure of standard random initialization to improve prediction accuracy.Small random noise can subsequently break ties among feature maps with a common predecessor.
  • Results: ≈64C^2 parameters characterize the basic module, which experiments show can increase dense prediction accuracy quantitatively and qualitatively.The result is notable because the module contains relatively few parameters.
  • Larger context network: A larger context network expands the number of feature maps in deeper layers and generalizes the initialization scheme to differing layer widths.The generalized scheme assumes C divides the feature-map counts in consecutive layers.

4 FRONT END

The front-end adapts VGG-16 for dense prediction by removing classification components that reduce resolution, producing 21 feature maps from a color image. This simplified module is more accurate than FCN-8s and DeepLab, exceeding DeepLab+CRF without using a CRF.

  • Architecture: The module adapts VGG-16 for dense prediction, removes the final two pooling and striding layers, and outputs C = 21 feature maps from a color image.Convolutions in subsequent layers were modified after removing those layers.
  • Architecture: Removing pooling layers simplified the network and made it more accurate than retaining them or replacing striding with dilation.The design also removes padding from intermediate feature maps.
  • Training: The module was trained on augmented Pascal VOC 2012 training data for 60K iterations using SGD with mini-batch size 14, learning rate 10^-3, and momentum 0.9.VOC-2012 validation images were excluded from training.
  • Results: The front-end outperformed both FCN-8s and DeepLab by more than 5 percentage points on the VOC-2012 test set.The comparison used models trained by the original authors and evaluated their test-set accuracy.
  • Results: 67.6% vs. 66.4%: the simplified front-end exceeded DeepLab+CRF leaderboard accuracy on the VOC-2012 test set without using a CRF.The reported comparison favors the simplified front-end module.

5 EXPERIMENTS

Experiments show that the dilated-convolution context module consistently improves semantic-segmentation accuracy across architectures and yields strong VOC-2012 results. The front end alone achieves high mean IoU, while the context module provides further gains and can outperform DeepLab-CRF-COCO-LargeFOV.

  • Implementation: The implementation is based on Caffe, and its dilated-convolution implementation is part of the standard Caffe distribution.The passage describes the implementation status rather than an accuracy result.
  • Front-end training and evaluation: The front end is trained using VOC-2012 and Microsoft COCO images, followed by VOC-2012-only fine-tuning.Training uses SGD with mini-batch size 14 and momentum 0.9; the procedure includes 100K, 40K, and 50K iteration stages with learning rates 10^-3, 10^-4, and 10^-5, respectively.
  • Front-end training and evaluation: 69.8% mean IoU on VOC-2012 validation and 71.3% mean IoU on the test set are achieved by the front-end module alone.These results exclude both the context module and structured prediction.
  • Controlled evaluation of context aggregation: The context module improves accuracy in all three evaluated semantic-segmentation configurations, with the Large module producing larger gains than the Basic module.The results also indicate synergy between context aggregation and structured prediction because the context module helps both with and without subsequent structured prediction.
  • Evaluation on the test set: On the Pascal VOC 2012 test set, the Large context module provides a significant accuracy boost over the front end and, alone, outperforms DeepLab-CRF-COCO-LargeFOV.The test-set results were submitted to the Pascal VOC 2012 evaluation server.

6 CONCLUSION

The paper argues that high-resolution operation and dilated convolutions are well suited to dense prediction because they expand receptive fields without sacrificing resolution or coverage. It presents the context network as a step toward dedicated, potentially fully dense architectures while acknowledging substantial room for future advances.

  • High-resolution operation throughout the network is described as both feasible and desirable for producing high-resolution dense predictions.
  • Dilated convolutions expand the receptive field without losing resolution or coverage, making them particularly suited to dense prediction.
  • The proposed network structure reliably increases accuracy when plugged into existing semantic segmentation systems.
  • The work points toward dedicated dense-prediction architectures, with future end-to-end dense training potentially enabling architectural simplification and unification.
  • State-of-the-art semantic segmentation systems still leave significant room for future advances, and the authors plan to release code and trained models.

APPENDIX A URBAN SCENE UNDERSTANDING

This appendix evaluates urban scene understanding on CamVid, KITTI, and Cityscapes using mean IoU, with training restricted to the training set and no structured prediction. It also describes shared training and context-module configurations across datasets.

  • Experimental setup: Experiments cover CamVid, KITTI, and Cityscapes, using mean IoU as the accuracy measure and excluding conditional random fields or other structured prediction.The model is trained only on each dataset’s training set, even when validation data are available.
  • Training procedure: Training uses stochastic gradient descent with mini-batches of 8 randomly sampled 628×628 crops from reflection-padded images.The learning rate is 10−4 and momentum is 0.99; intermediate layers use no padding.
  • Context modules: The dataset-specific context modules derive from the “Basic” network, use C channels per layer, and preserve input-response map size through padding.For Cityscapes, C = 19; the number of context-module layers depends on dataset image resolution.

A.1 CAMVID

On CamVid, the 8-layer Dilation8 network is trained with a staged front-end and joint-training procedure, and it outperforms prior semantic-segmentation methods on the test set.

  • A.1 CAMVID: The CamVid split contains 367 training, 100 validation, and 233 test images across 11 semantic classes, downsampled to 640×480.These are the dataset configuration used for evaluation.
  • A.1 CAMVID: Dilation8 denotes the complete convolutional network comprising the front-end and an 8-layer context module.The naming reflects the context module’s depth.
  • A.1 CAMVID: Training first runs the front-end for 20K iterations, then jointly trains the complete model on 852 × 852 crops with batch size 1, learning rate 10^-5, and momentum 0.9.The joint-training stage uses the specified crop size and optimization settings.
  • A.1 CAMVID: On the CamVid test set, Dilation8 outperforms ALE, SuperParsing, Liu and He, SegNet, and DeepLab-LargeFOV.The comparison is reported in Table 5 against the listed prior methods.

A.2 KITTI

On KITTI, the study uses 100 training images and 46 test images at 1226×370 resolution, adapting the context module to seven layers as Dilation7. Dilation7 outperforms prior work on semantic segmentation.

  • Dataset and architecture: The KITTI split contains 100 training images and 46 test images from the visual odometry/SLAM dataset, with resolution 1226×370.Because the vertical resolution is small, Layer 6 is removed, producing a seven-layer context module.
  • Dataset and architecture: The resulting complete network, combining the front-end and context module, is called Dilation7.
  • Training: The front-end is trained for 10K iterations, followed by 20K iterations of joint front-end and context-module training.Joint training uses 900×900 crops and momentum 0.99; other parameters match the CamVid setup.
  • Results: Dilation7 achieves higher semantic-segmentation accuracy on KITTI than the prior work compared in Table 6.The comparisons include Ros et al. (2015) and DeepLab-LargeFOV from Chen et al. (2015a).

A.3 CITYSCAPES

On Cityscapes, the authors adapt the context network to high-resolution images with two additional dilated layers, train the resulting Dilation10 model in three stages, and report that it outperformed prior models.

  • Architecture: Dilation10 adds post-Layer 6 dilation-32 and dilation-64 layers, producing a 10-layer context module for Cityscapes’ 2048×1024 images.The dataset contains 2975 training, 500 validation, and 1525 test images.
  • Training: The network uses three training stages: front-end pretraining, context training on whole images, and joint training on padded image halves.The stages run for 40K, 24K, and 60K iterations, respectively; the latter two use learning rates 10^-4 and 10^-5.
  • Results: Dilation10 outperformed all prior models in the Cityscapes evaluation reported by Cordts et al. (2016).Kundu et al. (2016) used Dilation10 as a unary classifier and further increased accuracy with structured prediction.
Loading 1511.07122v3…