Source-linked AI summary
How Much Position Information Do Convolutional Neural Networks Encode?
Md Amirul Islam, Sen Jia, Neil D. B. Bruce
TL;DR
The paper asks whether CNNs encode absolute position despite relying on local filters. It probes frozen CNN representations with PosENet across position-map tests and finds strong recoverable positional information, while identifying zero-padding and borders as likely anchors and noting semantic interference and scope limitations.
Problem
The paper investigates whether CNNs, whose local filters have limited spatial extent, implicitly encode absolute position needed for position-dependent vision tasks.
Method
The authors attach a trainable PosENet readout to frozen pretrained CNN features and evaluate predictions of gradient-like position maps using correlation and pixel-error metrics.
Results
Experiments show that commonly used CNN architectures contain strong recoverable absolute position information, including without semantic cues, with larger receptive fields and nonlinear readouts improving extraction.
Takeaways & Limitations
The findings support joint encoding of semantic content and absolute position, with zero-padding and image borders identified as anchors from which spatial information is derived and propagated.
Takeaways & Limitations
The reported analysis focuses on natural images, the PASCAL-S dataset, and three representative position patterns after observing low performance on repeated patterns.
Abstract
from arXiv · showhide
In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. An implication of this is that a filter may know what it is looking at, but not where it is positioned in the image. Information concerning absolute position is inherently useful, and it is reasonable to assume that deep CNNs may implicitly learn to encode this information if there is a means to do so. In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks. A comprehensive set of experiments show the validity of this hypothesis and shed light on how and where this information is represented while offering clues to where positional information is derived from in deep CNNs.
1 INTRODUCTION
CNNs use local filters efficiently, but this raises the question of whether they encode absolute position despite being viewed as spatially agnostic. Cropping experiments and the paper’s hypothesis motivate testing whether CNNs learn both what and where objects are.
- Motivation: CNNs achieve strong results across vision tasks while using filters with limited spatial visibility.This efficiency reduces the number of weights compared with fully connected networks.
- Motivation: Although CNNs are often considered spatially agnostic, absolute position matters for tasks such as semantic segmentation and salient object detection.Relative spatial relationships have motivated the use of capsule and recurrent networks.
- Motivation: Cropping an image shifts salient regions even when the visual features remain unchanged.The observed shift is notable because CNN filters have limited spatial extent and no explicit position encoding is described.
- Hypothesis: The paper hypothesizes that deep neural networks implicitly encode both what objects are and where they are located.Position is presented as a useful cue for likely object locations, such as birds appearing in the sky.
2 POSITION INFORMATION IN CNNS
The paper probes hidden absolute position information by attaching PosENet to frozen CNN features and training it to predict gradient-like position maps. The method combines multi-scale feature readout with position-map supervision and evaluates whether the frozen representations contain recoverable spatial information.
- Problem formulation: The study predicts gradient-like maps whose pixel values represent absolute horizontal or vertical coordinates.Ground-truth maps are generated for supervision while the base CNN weights remain fixed.
- Position Encoding Network: PosENet combines a feedforward convolutional encoder with a position encoding module that reads multi-scale features and predicts absolute position.The encoder uses ResNet- or VGG-based feature extractors, while the position module transforms features into a position map.
- Training and probing: The encoder is frozen while only the position encoding module is trained to extract position information from existing representations.This design probes whether positional information is already present rather than learned by updating the base encoder.
- Position Encoding Network: The position encoding module aligns multi-scale feature maps spatially, concatenates them, and applies k × k convolutions with k ∈ {1, 3, 5, 7}.Most experiments use a single convolutional layer in the position encoding module.
- Training and probing: The experiments supervise horizontal, vertical, Gaussian, and repeated position patterns using generated gradient-like maps.These patterns test whether networks can recover absolute position along one or two axes and under repeated structures.
3 EXPERIMENTS
Experiments test whether pretrained CNNs encode absolute position by probing frozen multi-level features with PosENet, using gradient-like targets and SPC/MAE. Results support implicit positional encoding across architectures, while performance depends on architecture, repeated-pattern difficulty, receptive field, and the tested image setting.
- Experimental setup: The study trains PosENet probes on DUT-S and evaluates positional information on natural PASCAL-S images and synthetic patterns.Networks use ImageNet-pretrained architectures, resized 224×224 inputs, and aligned 28×28 multi-level features.
- Experimental setup: SPC measures Spearman correlation with the ground-truth position map, while MAE measures average pixel-wise prediction error.The paper notes that position encoding lacks a universal metric and therefore uses both measures.
- Existence of position information: Pretrained VGG and ResNet models yield position information through PosENet, with ResNet-based probes generally extracting it more effectively.The comparison is made across different ground-truth patterns and source images.
- Existence of position information: Standalone PosENet scores much lower, indicating that consistent position-map extraction occurs when the probe is coupled with a deep encoder rather than the input image alone.The authors interpret the randomized ground-truth setup as evidence against blindly fitting noise.
- Existence of position information: Repeated horizontal and vertical stripe patterns perform worse than other patterns because they require greater model complexity and lack direct correlation with absolute position.The paper focuses subsequent experiments on natural PASCAL-S images and H, G, and HS patterns.
- Analyzing PosENet: Increasing PosENet depth or kernel size improves positional readout, consistent with larger effective receptive fields helping resolve spatially distributed information.The layer ablation fixes kernels at 3 × 3, while the kernel ablation uses one layer; Figure 5 visualizes these effects.
4 WHERE DOES POSITION INFORMATION COME FROM?
The experiments identify zero-padding and image borders as important sources of CNN absolute position information. Removing padding harms position recovery and performance on position-dependent tasks, while semantic content can interfere with position-map prediction.
- 4.1 CASE STUDY: Zero-padding near image borders is proposed as a source of position information learned by CNNs.The study removes padding from VGG16 while retaining ImageNet-pretrained weights to test this hypothesis.
- 4.1 CASE STUDY: Figure 7 shows PosENet’s content-loss heatmaps concentrating larger losses around corners, whereas VGG and ResNet losses correlate more with semantic content.ResNet exhibits especially strong semantic interference, with highest losses associated with faces, people, cats, airplanes, and vases.
- 4.2 ZERO-PADDING DRIVEN POSITION INFORMATION: VGG without zero-padding achieves much worse F-measure and MAE than padded VGG on salient-object detection across the evaluated datasets.This result is reported as further validation that zero-padding is a key source of position information.
- 4.2 ZERO-PADDING DRIVEN POSITION INFORMATION: VGG with zero-padding significantly outperforms VGG without padding on semantic segmentation.The comparison uses VGG16 models trained with and without zero padding on PASCAL VOC 2012.
- 4.2 ZERO-PADDING DRIVEN POSITION INFORMATION: The study compares VGG models pretrained on ImageNet classification, salient-object detection, and semantic segmentation to examine task-dependent position encoding.The models are fine-tuned using the protocol described for the position-information experiments.
5 CONCLUSION
The paper finds that CNNs implicitly encode substantial absolute position information, with zero-padding and borders serving as anchors. Larger receptive fields and nonlinear readouts improve recovery, while semantic features can interfere with position encoding.
- 5 CONCLUSION: Experiments show that absolute position information is available to a strong degree in CNNs.The conclusion presents this as a fundamental CNN property previously unknown to date.
- 5 CONCLUSION: Larger receptive fields and nonlinear readouts further augment absolute-position recovery beyond a single-layer 3 × 3 PosENet.The baseline single-layer readout already recovers position information strongly.
- 5 CONCLUSION: Position recovery remains possible without semantic cues, while semantic interference suggests joint encoding of what and where.The paper points to zero-padding and borders as anchors from which spatial information is derived and propagated across the image.