Source-linked AI summary
On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location
Osman Semih Kayhan, Jan C. van Gemert
TL;DR
The paper questions the assumption that convolutional layers in modern CNNs are translation invariant and examines how they can encode absolute position through image boundary effects. It analyzes convolution and padding variants, evaluates location exploitation across architectures, and investigates removing spatial-location encoding. The reported results show improved translation behavior and stronger low-data performance, with benefits across image classification, patch matching, and video classification.
Problem
The paper addresses the gap between CNNs' assumed translation invariance and evidence that convolutional layers can exploit absolute spatial location.
Method
The paper analyzes boundary effects and convolution-padding variants, then evaluates absolute-location exploitation and methods that remove spatial-location encoding across architectures and applications.
Results
Removing absolute-location encoding improves data efficiency and performance in low-data settings across image classification, patch matching, and video classification experiments.
Takeaways & Limitations
Translation equivariance is a stronger visual inductive bias when convolutional layers cannot exploit absolute spatial location, particularly for smaller datasets.
Takeaways & Limitations
Using F-Conv instead of S-Conv adds computation; a ResNet-50 with F-Conv was 15% slower to train on ImageNet.
Abstract
from arXiv · showhide
In this paper we challenge the common assumption that convolutional layers in modern CNNs are translation invariant. We show that CNNs can and will exploit the absolute spatial location by learning filters that respond exclusively to particular absolute locations by exploiting image boundary effects. Because modern CNNs filters have a huge receptive field, these boundary effects operate even far from the image boundary, allowing the network to exploit absolute spatial location all over the image. We give a simple solution to remove spatial location encoding which improves translation invariance and thus gives a stronger visual inductive bias which particularly benefits small data sets. We broadly demonstrate these benefits on several architectures and various applications such as image classification, patch matching, and two video classification datasets.
1. Introduction
CNNs are designed to share weights across image locations, making convolution translation equivariant and subsequent position-independent pooling translation invariant. The paper challenges this assumption, showing that even a simple fully convolutional CNN can classify identical patches by absolute location through boundary effects.
- CNN inductive bias: Convolution shares learnable weights across locations, reducing parameters and making shifted objects produce equally shifted convolution outputs.Position-independent global average or maximum pooling can then make the network translation invariant.
- Challenge to invariance: A one-layer fully convolutional CNN perfectly classifies identical patches placed at the top-left versus bottom-right of an image.The setup uses a single 5x5 kernel, zero-padding, same-convolution, ReLU, global max pooling, SGD, and softmax loss.
- Mechanism: Boundary effects let convolutional filters encode absolute position by retaining outputs associated with specific locations.Finite image support requires handling non-existing values beyond the image boundary.
- Scope: Modern CNNs can exploit absolute spatial location even far from image boundaries because their filters have large receptive fields.The paper reports this behavior across current CNN architectures and applications.
- Proposed direction: Removing spatial-location encoding strengthens the visual inductive bias and benefits smaller datasets across multiple architectures and application domains.The demonstrated domains include image classification, patch matching, and video classification.
2. Related Work and Relevance
Prior work uses convolution, pooling, parameter sharing, augmentation, and transformation-aware representations to shape CNN behavior and improve robustness or data efficiency. This paper instead focuses on how boundary effects allow convolutional layers themselves to encode absolute position, with implications for cropping and location-biased visual data.
- Network architectures: Fully connected layers can learn location-specific feature-map weights, whereas fully convolutional architectures replace them with position-invariant global pooling.Modern examples using global average pooling include residual, Inception, DenseNet, and ResNeXt networks.
- Transformation robustness: CNN robustness to geometric transformations is addressed through augmentation, adversarial training, or representations designed for equivariance and invariance.Prior examples cover rotation, scale, and other transformations.
- Translation behavior: Translation equivariance can also be lost through pooling and subsampling, but this paper attributes location encoding to image boundary effects instead.The paper distinguishes its focus from pooling-based loss of equivariance.
- Boundary handling: Boundary handling in CNNs has traditionally aimed to minimize distortions or model out-of-boundary pixels, whereas this paper studies boundary effects as a source of location encoding.Related approaches include separate boundary filters, missing-value treatment, and circular convolutions.
- Location information: Absolute location information has been explicitly added for patch matching, generative modeling, semantic segmentation, and instance segmentation.The paper instead investigates how standard convolutional processing can encode such information implicitly.
- Visual inductive priors: Data efficiency can improve when visual priors tie parameters, share rotation responses, or impose structured filter bases.The paper positions removing absolute-position encoding as another way to strengthen translation-related inductive bias.
3. How boundary effects encode location
Boundary handling makes convolution treat some absolute input locations differently. These unequal filter applications can encode location, while circularly padded same convolution and full convolution equalize responses.
- Boundary handling: Finite-support images require padding or boundary-specific handling when convolution reaches outside the image.Zero padding supplies zeros; circular padding wraps values from the opposite side.
- Valid convolution: Valid convolution omits boundary-crossing applications, shrinking the output and leaving some input locations without the full filter.The output loses 2k pixels per dimension, and the count falls below 2k + 1 near boundaries.
- Same convolution: Same convolution preserves output size but zero padding still leaves boundary locations where the full filter is not applied.Circular padding instead wraps values around and equalizes responses across absolute positions.
- Full convolution: Full convolution applies every filter value across all image values, so no input locations are left out.Its output grows by k elements at each boundary in each dimension.
- Location encoding: A two-dimensional example shows V-Conv and zero-padded S-Conv can distinguish identical pixels by whether they lie at the top-left or bottom-right.Circularly padded S-Conv and F-Conv retain the pixel value in both positions.
- Location encoding: Across architectures, larger receptive fields extend absolute-location classification farther from image boundaries.BagNet-33 is limited closer to the boundary, whereas ResNet-18 and DenseNet-121 classify locations farther away; training from scratch reaches farthest.
4. Experiments
The experiments test how CNNs exploit absolute position, how border handling affects this behavior, and whether removing location encoding improves shift robustness and data efficiency. Across classification, patch matching, and video settings, F-Conv generally improves translation-related behavior, especially with limited data.
- 4.1. Exp 1: How far from the image boundary can absolute location be exploited?: ResNet-18 can classify absolute position 128 pixels from an image boundary, while BagNet-33 fails farther from the boundary.Scratch-trained models reach farther than pretrained or randomly initialized models, demonstrating that location exploitation extends beyond boundary-adjacent regions.
- 4.2. Exp 2: Border handling variants: S-Conv-circ and F-Conv perform identically on similar and dissimilar location-biased test sets, indicating translation invariance.F-Conv outperforms S-Conv-circ because circular padding introduces new content, whereas Full-zero padding avoids semantic artifacts.
- 4.3. Exp 3: Sensitivity to image shifts: F-Conv improves classification accuracy and output consistency across architectures under diagonal image shifts.The largest standalone gains are 3.6% in accuracy and 2.49% in consistency, while ResNet-34 with BlurPool and F-Conv gains 4.85% accuracy and 3.91% consistency over S-Conv.
- 4.3. Exp 3: Sensitivity to image shifts: ResNet-101 benefits least from F-Conv because its larger capacity can learn more varied location-sensitive filters.The smaller gap occurs for both classification and consistency.
- 4.4. Exp 4: Data efficiency: F-Conv improves data efficiency for both ImageNet classification and patch matching when training data is reduced.With 50 ImageNet samples per class, accuracy rises from 26.4% with S-Conv to 31.1% with F-Conv; patch-matching scores change from 0.145 to 0.083.
- 4.5. Exp 5: Small datasets: F-Conv continues improving video classification while reducing overfitting relative to S-Conv.In UCF101, F-Conv improves performance by 3.6% for the same architectures, and a 3D ResNet-18 shows 25.7% overfitting versus 35.69% with S-Conv.
5. Limitations and Conclusion
The paper concludes that convolutional layers can encode absolute spatial location through boundary effects, but removing this feature improves shift stability and low-data accuracy. The authors caution that location should remain when it is genuinely discriminative, while noting computational costs for some padding choices.
- Limitations: Using F-Conv instead of S-Conv adds computation, with a ResNet-50 training 15% slower on ImageNet.The paper states that circular padding has no extra cost relative to zero padding, while F-Conv has costs similar to replacing S-Conv with V-Conv.
- Limitations: Location encoding should not be removed when absolute spatial position is genuinely discriminative between classes.The paper distinguishes useful location information from spurious location correlations caused by limited data, such as centered subjects, bottom-positioned sofas, or upper-positioned sky.
- Conclusion: Convolutional layers can encode absolute spatial location through boundary effects, with large receptive fields making this available throughout images.The authors report that removing spatial location improves stability to image shifts and the visual inductive prior, increasing accuracy in low-data settings across several applications.
- Conclusion: Removing spatial location improves shift stability and low-data accuracy across ImageNet classification, patch matching, and two video classification datasets.