On the Relationship between Self-Attention and Convolutional Layers

Jean-Baptiste Cordonnier; Andreas Loukas; Martin Jaggi

Source-linked AI summary

On the Relationship between Self-Attention and Convolutional Layers

Jean-Baptiste Cordonnier, Andreas Loukas, Martin Jaggi

arXiv:1911.03584v2Nov 8, 2019cs.LGcs.CLcs.CVstat.ML

TL;DR

The paper asks whether learned self-attention processes images similarly to convolutional layers, since universality alone does not explain how models compute. It proves constructively that multi-head self-attention can express convolution and empirically finds grid-like local attention patterns, including with learned relative encoding. These findings support viewing attention-only vision models as combining convolution-like local behavior with content-based global attention.

Problem
The paper examines whether self-attention layers process images similarly to convolutional layers, because general universality results do not reveal how attention solves visual tasks.
Method
The paper combines a constructive expressivity proof with experiments examining attention patterns in attention-only image models and different positional encodings.
Results
A single multi-head self-attention layer with relative positional encoding can express any convolutional layer, while experiments show learned grid-like pixel attention and matching behavior for learned relative encoding.
Takeaways & Limitations
Self-attention can learn convolution-like local behavior and combine it with global, content-based attention, resembling a generalization of CNNs with learned kernel patterns.
Takeaways & Limitations
The construction extends to 1D convolution, but the paper does not empirically establish that 1D self-attention learns to convolve input sequences.

Abstract

from arXiv · show

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.

1 INTRODUCTION

The paper asks whether self-attention processes images like convolutional layers, beyond merely having the theoretical capacity to simulate them. It presents constructive theoretical evidence and experiments showing that attention can express convolution and learns grid-like local patterns.

Motivation: The question remains open because universality results establish capacity but do not explain how attention actually processes images.This motivates examining learned attention behavior rather than relying only on expressivity arguments.
Contributions: Self-attention layers can behave similarly to convolutional layers in vision models.The work combines theoretical and empirical evidence for this claim.
Contributions: A single multi-head self-attention layer with relative positional encoding can be re-parametrized to express any convolutional layer.The construction requires a sufficient number of heads.
Contributions: Attention-only architectures learn to attend to grid-like patterns around each query pixel, similarly to the theoretical construction.This behavior is observed in the first few layers.

2 BACKGROUND ON ATTENTION MECHANISMS FOR VISION

The background introduces self-attention, positional encodings, and their image adaptation, then contrasts absolute and relative position representations. Relative encoding makes attention scores depend on the shift between query and key pixels.

Self-attention: Self-attention maps input tokens to output representations using query, key, and value projections.The same formalism applies to sequences of pixels as well as words.
Positional encoding: Without positional information, self-attention is equivariant to token reordering, which is problematic when order matters.A positional encoding is added to each token or pixel representation before attention.
Multi-head attention: Multi-head self-attention uses separate query, key, and value matrices so heads can focus on different parts of the input.Head outputs are concatenated and projected to the final output dimension.
Attention for images: For images, each attention score associates a query pixel with a key pixel, and the input is represented as a width-by-height-by-channel tensor.The attention output is computed at each query pixel from attended pixel values.
Positional encoding: Relative positional encoding uses the position difference between query and key pixels rather than the key pixel’s absolute position.Consequently, attention scores depend only on the shift δ := k − q; relative position vectors are shared across layers and heads.

3 SELF-ATTENTION AS A CONVOLUTIONAL LAYER

A multi-head self-attention layer with relative positional encoding can simulate convolution by assigning heads to pixel shifts and learning corresponding filter matrices. The construction extends to stride, dilation, and specified channel constraints, with finite-precision considerations.

A multi-head self-attention layer with relative positional encoding can express any convolutional layer of kernel size √Nh × √Nh and min(Dh, Dout) output channels.
The constructive proof assigns each of Nh = K^2 heads to a distinct relative shift in the K × K kernel grid.Each head's attention concentrates on its assigned shift, while its learned matrix supplies the corresponding convolutional filter.
Stride is handled by appending fixed pooling, while matching convolutional boundary behavior requires zero-padding the input and cropping the output.
The construction supports dilated convolutions because heads can attend to arbitrary pixel shifts and form a dilated grid pattern.
For K = √Nh, the head matrices map one-to-one to the convolutional kernel matrices, making the self-attention output equivalent to convolution.
The theorem applies to min(Dh, Dout) output channels; when Dh < Dout, the construction cannot express every convolution with Dout channels.The authors recommend concatenating heads of dimension Dh to cover both channel configurations.
The proof uses a three-dimensional relative encoding, and finite precision permits a finite attention scale such as α = 46 to approximate hard attention.The idealized exact representation requires arbitrarily large α, while float32 arithmetic makes a finite value sufficient for the construction.

4 EXPERIMENTS

The experiments test whether self-attention learns convolution-like behavior in fully attentional image classifiers using quadratic and learned relative positional encodings. Across settings, attention heads form localized, grid-like patterns in early layers while deeper layers also capture larger or content-based dependencies.

4 EXPERIMENTS: The study evaluates six self-attention layers on CIFAR-10, comparing the models with a standard ResNet18 to validate meaningful classification.The authors state that achieving state-of-the-art performance is not the focus.
4.1 IMPLEMENTATION DETAILS: The ResNet converges faster, while learned embeddings with content-based attention are harder to train because they likely use more parameters.The authors believe the performance gap could be bridged but do not make that optimization the focus.
4.2 QUADRATIC ENCODING: After training with quadratic relative positional encoding, layer-4 heads attend to specific pixels arranged in a grid around each query pixel.This behavior confirms the authors’ intuition that self-attention can learn convolutional filters around the queried pixel.
4.2 QUADRATIC ENCODING: With quadratic positional encoding, early layers focus on local patterns, whereas layers 3–6 also attend to larger patterns farther from the query pixel.The heads do not overlap and appear arranged to maximize coverage of the input space.
4.3 LEARNED RELATIVE POSITIONAL ENCODING: Without content-based attention, learned positional encodings produce localized heads attending to individual pixels, alongside non-localized and long-range patterns.The positional-only experiment confirms the hypothesis for the first two layers and partially for the third.
4.3 LEARNED RELATIVE POSITIONAL ENCODING: With both positional and content-based attention, some layer-2 and layer-3 heads attend to fixed-shift pixels, reproducing a convolutional kernel’s receptive field.When the query pixel moves, these localized patterns follow it and remain at a constant shift, similarly to convolution.

5 RELATED WORK

Prior work compared transformers and CNNs mainly on performance, computational cost, and long-term dependencies. This paper addresses expressiveness by showing that a self-attention layer can encompass all convolutional filters.

Transformer and CNN comparisons have primarily examined performance, computational cost, and long-term dependencies.
Transformers have been shown to be Turing-complete, but this theoretical result is not informative for practitioners.
The paper reports being the first to show that a self-attention layer’s function class contains all convolutional filters.
Andreoli unified attention and convolution using a tensor-outer-product framework representing convolutional receptive fields with basis tensors.

6 CONCLUSION

The paper concludes that image self-attention can express any convolutional layer with sufficiently many heads. Fully attentional models also learn local, convolution-like behavior alongside global, content-based attention.

Self-attention layers applied to images can express any convolutional layer given sufficiently many heads.
Fully-attentional models combine local behavior resembling convolution with global attention based on input content.
The authors relate this learned kernel-pattern behavior to deformable convolutions.

A MORE EXAMPLES WITH CONTENT-BASED ATTENTION

The appendix presents additional attention visualizations for content-based self-attention. They cover averaged attention across 100 test images and single-image examples at different query pixels.

Figure 7 averages attention probabilities across 100 test images for a model with 6 layers and 9 heads.
Figure 7 arranges attention probabilities with layers as rows and heads as columns.
The visualizations use learned relative positional encoding and content-content-based attention, with the query pixel marked by a black square.
Figures 8 and 9 show single-image attention when the query pixel lies on a frog head or horse head.
Figure 10 shows single-image attention when the query pixel lies on a building in the background.

B HYPER-PARAMETERS USED IN OUR EXPERIMENTS

This section identifies Table 2 as presenting the parameters of the self-attention network.

Table 2 presents the parameters used for the self-attention network.

C POSITIONAL ENCODING REFERENCES

The section reformulates convolution and multi-head self-attention as factorizations of position-dependent linear transformations, establishing when attention can represent convolution exactly. The key condition is that the convolutional receptive-field structure lies within the span of the attention probabilities.

Expressivity: A multi-head self-attention layer with relative positional encoding can be re-parameterized to express any convolutional layer.The construction uses sufficient heads, head dimension Dh ≥ Dout, and a convolutional kernel with K × K support.
Convolution factorization: Convolutional weights at query position q have restricted support over the K × K receptive field and factor as W convEq.W conv stores vectorized kernel weights, while Eq selects the indexed pixel shifts for q.
Attention factorization: Self-attention yields an analogous factorization V SAq = W SAAq, where W SA collects head transformations and Aq contains attention probabilities.The multi-head formulation makes the contribution of each head explicit through its value and output projections.
Representation condition: Exact equality for every input requires and is guaranteed by row(Eq) ⊆ row(Aq), allowing Eq = ΦAq and W SA = W convΦ.If a row of Eq lies outside row(Aq), no corresponding decomposition exists for arbitrary convolutional weights.

E GENERALIZED QUADRATIC POSITIONAL ENCODING

The generalized quadratic positional encoding models attention over pixel shifts with non-isotropic Gaussian distributions. Each head learns a center and covariance, enabling attention over elongated or otherwise anisotropic pixel groups.

Gaussian parameterization: Non-isotropic Gaussian attention parameterizes each head by a center of attention Δ and covariance matrix Σ over pixel positions.The covariance parameterization is designed to produce positive semi-definite covariance matrices.
Quadratic encoding: The attention coefficient is expressed as a dot product between a head target vector v and relative position features rδ.The relative shift is δ = k − q, and rδ contains first- and second-order combinations of its pixel coordinates.
Evaluation: Training tests whether self-attention learns non-isotropic pixel groups that are not typically produced by convolutional patterns.Centers are initialized with Δ^(h) ∼ N(0, 2I2), while the inverse square-root covariance starts near the identity.
Degenerate heads: Degenerated heads may attend to very thin pixel stripes or uniformly across all pixels when the inverse covariance is nearly singular or nearly zero.The study prunes heads with largest eigen-values below 10^-5 or with a large condition number.

F INCREASING THE NUMBER OF HEADS

The increased-head experiment examines attention patterns across six self-attention layers. It finds localized heads early and less-localized heads increasingly often in higher layers, with localized heads behaving like convolution.

Experimental setup: Increasing the architecture from 9 to 16 heads was tested as an additional experimental condition.
Layer-wise patterns: Localized heads occur more frequently in the first few layers, while less-localized heads become more common at higher layers.The reported patterns are shown for 16 attention heads across six self-attention layers.
Convolution-like behavior: Localized heads attend to nearly individual pixels and make the self-attention layer behave similarly to convolutional layers.