Source-linked AI summary
Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
Wenjie Luo, Yujia Li, Raquel Urtasun, Richard Zemel
TL;DR
The paper addresses how much each input pixel influences a deep CNN output, a question that matters when predictions require broad image context. It develops a theory and empirical analysis of effective receptive fields across network factors. The impact is asymptotically Gaussian, the effective receptive field occupies only part of the theoretical field, and common initialization can start it small before training enlarges it.
Problem
Receptive fields must cover relevant image regions for visual tasks, but the distribution and practically influential portion of a CNN unit’s theoretical receptive field require characterization.
Method
The paper mathematically characterizes pixel impact within deep CNN receptive fields and studies empirical effects of kernels, activations, dropout, sub-sampling, skip connections, and initialization.
Results
The impact distribution is asymptotically Gaussian, and the effective receptive field occupies only a fraction of the full theoretical receptive field; empirical results support the theory.
Takeaways & Limitations
The effective receptive field provides a new way to understand deep CNNs and motivates potential methods for increasing its size when large receptive fields are required.
Takeaways & Limitations
The analysis for negative kernel weights is more difficult and is left to future work, although empirical implications still apply reasonably well in such networks.
Abstract
from arXiv · showhide
We study characteristics of receptive fields of units in deep convolutional networks. The receptive field size is a crucial issue in many visual tasks, as the output must respond to large enough areas in the image to capture information about large objects. We introduce the notion of an effective receptive field, and show that it both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field. We analyze the effective receptive field in several architecture designs, and the effect of nonlinear activations, dropout, sub-sampling and skip connections on it. This leads to suggestions for ways to address its tendency to be too small.
1 Introduction
Deep CNN units respond only to local input regions, making receptive-field size important for visual predictions. The paper argues that impact within the theoretical receptive field is uneven, with a Gaussian-like effective region occupying only part of it.
- A unit’s receptive field is the input region that can affect its value, unlike a fully connected unit’s dependence on the entire input.
- Large receptive fields are critical when outputs must incorporate all relevant image information, especially in dense prediction tasks such as semantic segmentation, stereo, and optical flow.
- Stacking layers increases theoretical receptive-field size linearly, while sub-sampling increases it multiplicatively.
- Pixels within a receptive field do not contribute equally: central pixels have more propagation paths and larger backward gradients than outer pixels.
- The impact distribution is often provably Gaussian, so the effective receptive field occupies only a fraction of the theoretical receptive field.
- Common random initializations can produce a small effective receptive field that grows during training, potentially indicating an initialization bias.
2 Properties of Effective Receptive Fields
The paper defines the effective receptive field through input-output impact and shows that, across analyzed CNN settings, this impact is approximately Gaussian and occupies only part of the theoretical receptive field.
- Definition and measurement: The effective receptive field is the region of input pixels with non-negligible impact on a chosen output unit, measured by the partial derivative of output with respect to each input pixel.Because this derivative can depend on the input, results are often expressed as expectations over the input distribution.
- Linear networks: In deep linear CNNs with uniform kernels, repeated convolution produces binomial or extended binomial coefficients whose 2D impact distribution is Gaussian-shaped.The 2D result follows by decomposing the all-ones kernel into products of 1D convolutions.
- Random weights: For random weights, the variance of the input gradients retains a Gaussian shape, with an additional C^n factor that does not change relative impact within the receptive field.The random-weight analysis assumes independent weights with zero mean and variance C, and gradient-weight independence in linear networks.
- Non-uniform kernels: The Gaussian impact decays squared-exponentially away from the receptive-field center, and one standard deviation is used as an approximate ERF radius.The decay rate is determined by the Gaussian variance.
- Non-uniform kernels: As more convolutional layers are stacked, the theoretical receptive field grows linearly while the ERF-to-theoretical-field ratio shrinks at O(1/√n).For uniform weighting, the ERF size itself grows linearly with kernel size k.
- Nonlinear activation functions: For nonlinear activations, the analyzed variance case reduces to the uniform-weight analysis up to a constant factor, while Sigmoid and Tanh become difficult to analyze as weights grow during training.The linear analysis is used for Sigmoid and Tanh at initialization, when weights are usually small.
- Architectural effects: Dropout preserves the Gaussian ERF shape, whereas subsampling and dilated convolutions increase receptive-field size quickly and skip-connections make ERFs smaller.These architectural effects are analyzed for standard CNN approaches.
3 Experiments
The experiments verify the effective receptive field (ERF) theory and examine how architecture and training affect ERF size and shape. ERFs are near-Gaussian in tested networks, while subsampling, dilation, and training substantially enlarge them.
- Experimental setup: A unit gradient at the output is back-propagated to the input to compute ERFs across artificially constructed and trained CNNs.The experiments use random inputs and proper random initialization for random-weight networks.
- Verifying theoretical results: Uniform and random convolutional kernels produce nearly Gaussian ERFs, including with ReLU, although nonlinearity makes the distribution somewhat less Gaussian.Figure 1 fixes the kernel size at 3 × 3 and compares uniform, random, and random-plus-ReLU networks.
- Verifying theoretical results: ERF size grows approximately as √n while its ratio to the theoretical receptive field shrinks approximately as 1/√n with convolutional depth.The fitted log-domain slopes are 0.56 for ERF size and -0.43 for ERF ratio.
- Verifying theoretical results: Subsampling and dilated convolution both significantly increase the ERF relative to a 15-layer dense-convolution baseline.Dilation factors 2, 4, and 8 produce a rectangular ERF shape typical of dilated convolutions.
- How the ERF evolves during training: In trained ResNets, the ERF grows from its initial state for both CIFAR-10 classification and CamVid semantic segmentation.The CamVid model's ERF increases from about diameter 100 to almost 150, despite a theoretical receptive field of 505 × 505.
4 Reduce the Gaussian Damage
The paper considers initialization and architectural changes to enlarge the effective receptive field, which otherwise occupies only a small portion of the theoretical field. A proposed initialization diffuses weight concentration outward, but its benefits are inconsistent and only partial.
- The effective receptive field occupies only a small portion of the theoretical receptive field, creating a problem for tasks requiring broad context.
- New Initialization: The proposed initialization assigns smaller scales to central kernel weights and larger scales to outer weights to diffuse influence toward the periphery.
- New Initialization: A corner-weight optimization maximizes effective receptive field variance, but its many zero weights make learning slow, so a softer version is preferred.
- New Initialization: A few CIFAR-10 trials achieved a 30% training speed-up over standard initializations, but the overall benefit was not always significant.
- New Initialization: Changing initialization weights cannot remove the Gaussian distribution of the effective receptive field, so the proposal solves the problem only partially.
- Architectural changes: Architectural alternatives include sparse connections to larger lower-layer areas, extending beyond grid-like dilated convolution.
5 Discussion
The discussion connects Gaussian effective receptive fields to biological vision and prior CNN initialization and visualization work. The analysis suggests retained position information and naturally foveated representations in convolutional networks.
- Connection to biological neural networks: Because the effective receptive field grows slower than the theoretical field, deep CNNs preserve substantial local information and may retain position information.
- Connection to biological neural networks: The Gaussian receptive-field structure suggests that convolutional networks may automatically form a foveal representation with higher resolution near the center.
- Connection to previous work on CNNs: Earlier CNN analyses studied variance evolution and proposed initialization schemes designed to keep variance nearly unchanged across layers.
- Connection to previous work on CNNs: Visualization studies used natural-image priors, deconvolutional networks, receptive fields, and gradient ascent to relate image pixels, activations, and localization.
6 Conclusion
The paper establishes that impact within deep CNN receptive fields is asymptotically Gaussian and that the effective receptive field occupies only a fraction of the theoretical field. Empirical results support these theoretical findings, while future work remains on controlling effective receptive fields in practice.
- The impact distribution within deep CNN receptive fields is asymptotically Gaussian, and the effective receptive field occupies only a fraction of the theoretical field.
- Empirical results echoed the established theory, motivating further study of the factors that affect effective receptive fields and how to control them.