Source-linked AI summary
Dilated Residual Networks
Fisher Yu, Vladlen Koltun, Thomas Funkhouser
TL;DR
Convolutional networks can lose spatial detail through progressive downsampling, limiting classification and transfer to detailed scene-understanding tasks. The paper replaces selected subsampling with dilation in residual networks, preserving receptive fields while increasing output resolution. DRNs improve image classification and downstream localization and segmentation, with further gains from degridding, while higher resolution carries memory costs.
Problem
Progressive downsampling leaves tiny feature maps with limited spatial acuity, which can hinder natural-image classification and transfer to detailed scene-understanding tasks.
Method
The paper converts ResNets into DRNs by replacing selected interior subsampling layers with dilation, preserving receptive fields and higher-resolution outputs.
Results
DRNs improve image classification over non-dilated counterparts without added depth or complexity, and further improve through degridding and downstream applications.
Takeaways & Limitations
High-resolution DRN activations can support weakly-supervised localization without fine-tuning and dense prediction with performance exceeding deeper, higher-capacity baselines.
Takeaways & Limitations
Increasing feature-map resolution raises memory consumption fourfold for every twofold increase in each spatial dimension, limiting full-resolution operation.
Abstract
from arXiv · showhide
Convolutional networks for image classification progressively reduce resolution until the image is represented by tiny feature maps in which the spatial structure of the scene is no longer discernible. Such loss of spatial acuity can limit image classification accuracy and complicate the transfer of the model to downstream applications that require detailed scene understanding. These problems can be alleviated by dilation, which increases the resolution of output feature maps without reducing the receptive field of individual neurons. We show that dilated residual networks (DRNs) outperform their non-dilated counterparts in image classification without increasing the model's depth or complexity. We then study gridding artifacts introduced by dilation, develop an approach to removing these artifacts (`degridding'), and show that this further increases the performance of DRNs. In addition, we show that the accuracy advantage of DRNs is further magnified in downstream applications such as object localization and semantic segmentation.
1. Introduction
Convolutional networks often compress images into tiny feature maps, losing spatial detail that may matter for natural-image classification and downstream scene understanding. DRNs preserve higher resolution through dilation and improve classification and downstream accuracy without added depth or complexity.
- Typical classifiers reduce images to 7×7 feature maps that retain little spatial information.
- Preserving spatial acuity may retain small, thin objects and relative configurations important for understanding complex natural scenes.
- Lost resolution handicaps transfer to detailed applications, motivating post-hoc up-convolutions and skip connections.
- DRNs replace selected interior subsampling layers with dilation, increasing output resolution without increasing depth or model complexity.
- 28×28 DRN outputs preserve thumbnail-level scene structure, while degridding further improves accuracy by addressing dilation artifacts.
- DRNs improve downstream weakly-supervised localization and semantic segmentation, including a 42-layer DRN exceeding ResNet-101 on Cityscapes by more than 4 percentage points.
2. Dilated Residual Networks
DRNs preserve higher-resolution representations by removing selected striding and using dilation to retain receptive fields. The construction keeps the original ResNet’s layers and parameters while balancing spatial detail against memory costs.
- The design preserves spatial resolution so dense output signals can retain information about smaller, less salient objects.
- The construction starts from five groups of convolutional layers, each beginning with a strided downsampling layer.
- Simply removing striding increases resolution but shrinks subsequent receptive fields, reducing contextual information available for prediction.
- Dilated convolutions compensate for this shrinkage so higher-layer units retain the original model’s receptive fields.
- The conversion removes striding in G4 and G5, then applies dilation factors of 2 and 4 to compensate for the resulting receptive-field reductions.
- The converted DRN preserves the original ResNet’s layer and parameter counts while changing downsampling from 32× to 8× and producing 28×28 G5 outputs.
- The construction stops at 8× downsampling because higher resolution increases memory consumption fourfold for each twofold increase per spatial dimension.
3. Localization
DRNs can be reconfigured for localization by producing dense, high-resolution class activation maps directly from a classification model. This requires no additional training or parameter tuning.
- Direct transfer: A DRN trained for image classification can directly produce dense pixel-level class activation maps for localization and segmentation.The classification model already produces high-resolution output maps, avoiding post-hoc architectural changes and retraining.
- Activation maps: Removing global average pooling and connecting the final 1×1 convolution to G5 yields 28×28 activation maps with one map per classification category.A softmax converts each pixel’s prediction scores into probability distributions.
- Direct transfer: Localization reconfiguration removes the pooling operator and produces n activation maps at resolution w × h without training or parameter tuning.
- Activation maps: Each activation-map pixel contains the probability that the object observed there belongs to its corresponding category.
- Comparison with ResNets: DRNs produce higher-resolution activation maps than corresponding ResNets, improving their suitability for weakly supervised object localization.
4. Degridding
Dilation can introduce gridding artifacts when feature-map frequencies exceed the dilated convolution’s sampling rate. The degridding scheme progressively modifies early and late network components to reduce these artifacts.
- Artifact formation: Gridding artifacts arise when a feature map contains higher-frequency content than the sampling rate of a dilated convolution.A single active input pixel passed through a 2-dilated convolution produces a grid pattern in the output.
- Architecture changes: DRN-A directly dilates the ResNet architecture, while DRN-B and DRN-C progressively modify it to remove gridding artifacts.DRN-B replaces early max pooling and adds late residual blocks; DRN-C additionally removes residual connections from some added blocks.
- Early-layer modification: Replacing early max pooling with convolutional filters reduces high-frequency activations that can propagate through later layers and exacerbate gridding artifacts.The first degridding stage modifies the early layers and compares feature maps from DRN-A-18 and DRN-B-26.
- Late-layer modification: Removing residual connections from the added levels prevents artifacts from level 6 from propagating through levels 7 and 8.This change produces the final DRN-C construction.
5. Experiments
Experiments show that DRNs improve ImageNet classification and transfer effectively to weakly supervised localization and Cityscapes semantic segmentation. Degridding further improves performance, especially for applications requiring detailed spatial analysis.
- Image Classification: DRN-A models outperform corresponding ResNets in ImageNet classification despite identical depth and capacity.DRN-A-18 and DRN-A-34 improve 1-crop top-1 accuracy by 2.43 and 2.92 percentage points, respectively.
- Image Classification: DRN-A-50 exceeds ResNet-50 by more than one percentage point in 1-crop top-1 accuracy.The corresponding ResNet-152 over ResNet-101 error reduction is 0.3 percentage points.
- Image Classification: Degridding substantially improves DRN accuracy: DRN-C-26 matches DRN-A-34, while DRN-C-42 matches DRN-A-50.These DRN-C models are derived from shallower DRN-A models, although the degridding procedure increases depth and capacity.
- Object Localization: DRNs outperform corresponding ResNets in weakly-supervised object localization, and DRN-C-26 exceeds both DRN-A-50 and ResNet-101 despite much lower depth.Classification networks are used directly for localization without fine-tuning.
- Semantic Segmentation: All presented DRN models outperform the reported ResNet-101 Cityscapes baseline, with DRN-C-42 exceeding it by more than 4 percentage points despite 2.4 times lower depth.The baseline achieves a reported mean IoU of 66.6; DRN-C-26 exceeds it by more than one percentage point.
- Semantic Segmentation: DRN-C-26 produces cleaner semantic-segmentation predictions than DRN-A-50 by reducing visible gridding artifacts.The comparison is shown qualitatively on Cityscapes in Figure 8.
6. Conclusion
The paper preserves high spatial resolution through the final output layers, improving classification and further improving accuracy by alleviating dilation-induced gridding artifacts. These informative activations transfer directly to localization and segmentation, supporting detailed analysis of complex natural images.
- Preserving high spatial resolution through the final output layers improves image classification accuracy over state-of-the-art models.
- Alleviating gridding artifacts introduced by dilation increases accuracy further.
- Informative output activations support weakly-supervised object localization without fine-tuning and dense prediction through semantic segmentation.
- Dilated residual networks can serve as starting points for image-analysis tasks involving complex natural images and detailed scene understanding.