Source-linked AI summary
Exploring Self-attention for Image Recognition
Hengshuang Zhao, Jiaya Jia, Vladlen Koltun
TL;DR
Image recognition models have traditionally relied on convolution, whose fixed local aggregation has important limitations. This paper evaluates pairwise and patchwise self-attention as alternative building blocks and finds comparable or better performance, including substantially higher performance from patchwise models, alongside potential robustness and generalization benefits.
Problem
Convolution is effective for image recognition but lacks rotation invariance, scales parameter count with kernel footprint, and cannot adapt neighborhood aggregation to content.
Method
The paper evaluates pairwise and patchwise self-attention networks as fully self-attention-based image recognition models and compares them with convolutional ResNet models.
Results
Pairwise self-attention networks match or outperform convolutional counterparts, while patchwise models substantially outperform convolutional baselines; SAN15 reaches 78% top-1 accuracy versus ResNet50's 76.9%.
Takeaways & Limitations
Self-attention provides an alternative route to comparable or higher discriminative power, with structural properties including permutation- and cardinality-invariance and potential robustness and generalization benefits.
Takeaways & Limitations
A controlled comparison with Ramachandran et al. was limited because their implementation was unavailable and several subtle configuration differences could affect results.
Abstract
from arXiv · showhide
Recent work has shown that self-attention can serve as a basic building block for image recognition models. We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. Our pairwise self-attention networks match or outperform their convolutional counterparts, and the patchwise models substantially outperform the convolutional baselines. We also conduct experiments that probe the robustness of learned representations and conclude that self-attention networks may have significant benefits in terms of robustness and generalization.
1. Introduction
The paper asks whether convolutional networks are a local optimum for image recognition and explores self-attention as an alternative. It compares pairwise and patchwise self-attention, finding both effective, with patchwise models substantially outperforming convolutional baselines.
- Convolution lacks rotation invariance, requires more parameters for larger kernels, and uses stationary neighborhood aggregation that cannot adapt to content.
- Self-attention is explored as a potentially complementary alternative to convolution for image recognition.
- Pairwise self-attention is a permutation- and cardinality-invariant set operator whose footprint can expand without increasing parameter count.
- Patchwise self-attention can identify specific locations and is strictly more powerful than convolution.
- 78% top-1 accuracy for SAN15 versus 76.9% for ResNet50, with a 37% lower parameter and FLOP count.
2. Related Work
Convolutional networks became dominant in computer vision, while self-attention expanded from language into vision and other visual tasks. Earlier vision applications generally complemented convolution, whereas later work explored local attention to reduce computational costs and enable broader use.
- Convolutional networks: Convolutional networks became mainstream for image recognition after successful ImageNet applications and subsequent architectural advances.They also underpin models for other computer vision tasks.
- Self-attention: Self-attention, first broadly successful in machine translation and natural language processing, has been applied to image recognition, synthesis, captioning, and video prediction.
- Self-attention: Until recently, computer-vision self-attention primarily supplemented, modulated, or combined with convolution rather than replacing it.One cited study found its global self-attention insufficiently powerful to replace convolutions entirely.
- Self-attention: Dynamic filter networks generalized convolutions but incurred substantial memory and computational costs, limiting scaling to high-resolution images and larger datasets.
- Local attention: Restricting self-attention to local patches, such as 7×7 pixels, limits memory and computation and facilitates applying it throughout a network.This contrasts with earlier constructions using global attention over whole feature maps.
3. Self-attention Networks
The paper separates feature aggregation from feature transformation and builds fully self-attentional networks around two aggregation operators: pairwise and patchwise self-attention. These operators are organized into residual-style architectures corresponding roughly to ResNet baselines.
- Construction: Self-attention networks decouple feature aggregation from feature transformation, focusing self-attention on aggregation while pointwise perceptrons perform transformation.Convolutional layers combine both functions, whereas the proposed construction uses feature-wise aggregation followed by per-vector linear and nonlinear processing.
- Pairwise Self-attention: Pairwise self-attention aggregates transformed features β(xj) using adaptive weight vectors α(xi, xj) over a local footprint R(i).The relation function δ represents xi and xj, while γ maps that relation into a vector usable with β(xj).
- Pairwise Self-attention: Pairwise attention uses relation functions including summation, subtraction, Hadamard product, concatenation, and dot product, with different output dimensionalities.Summation, subtraction, and Hadamard product preserve the transformation dimensionality; concatenation doubles it, while dot product produces a scalar.
- Patchwise Self-attention: Patchwise self-attention computes a tensor of location-specific weight vectors from all feature vectors in the local patch xR(i).Unlike pairwise attention, it can index locations individually and intermix information across the footprint, so it is neither permutation-invariant nor cardinality-invariant.
- Network Architectures: The self-attention block uses parallel attention-weight and feature-transformation streams, reduces channel dimensionality in both, combines outputs with a Hadamard product, and expands back to C.The left stream computes α through δ and γ; the right stream applies β.
- Network Architectures: SAN10, SAN15, and SAN19 are fully self-attentional architectures roughly corresponding to ResNet26, ResNet38, and ResNet50.The networks stack self-attention blocks across five stages, with transition layers reducing spatial resolution and expanding channel dimensionality.
4. Comparison
Table 2 compares convolution, scalar attention, and the paper’s vector and patchwise operators by how their weights adapt to image content, spatial position, and channels.
- Table 2 summarizes these differences: the paper’s operators adapt attention weights across both spatial dimensions and channels.
- Convolution uses fixed kernel weights independent of image content, although weights can vary across channels.
- Scalar attention adapts aggregation weights across locations based on image content, but its weights remain shared across all channels.
- Vector attention generalizes scalar attention by producing vector-valued relations and attention weights that vary across spatial and channel dimensions.Its mappings can incorporate position encoding channels and reduce dimensionality for computational efficiency.
- Patchwise operators generalize convolution while retaining parameter and FLOP efficiency, with weight vectors informed by each operator footprint.
5. Experiments
Experiments on ImageNet compare self-attention networks with convolutional baselines and test architectural choices, robustness, and generalization. Pairwise models match or outperform comparable convolutions, while patchwise models achieve stronger results; self-attention also shows robustness advantages under distribution shifts and attacks.
- Experimental setup: ImageNet experiments compare self-attention networks with convolutional ResNet26, ResNet38, and ResNet50 baselines using single-crop validation accuracy.The dataset contains 1.28 million training images, 50K validation images, and 1000 classes.
- Comparison to convolutional networks: SAN15 reaches 78% top-1 accuracy versus 76% for ResNet38 and 76.9% for ResNet50, with 37% lower parameters and FLOPs than ResNet50.Patchwise SAN10 also outperforms ResNet38 with 40% fewer parameters and 41% fewer FLOPs.
- Controlled experiments: Pairwise relation functions using summation, subtraction, or Hadamard product outperform concatenation and dot product, indicating that vector self-attention outperforms scalar self-attention.For patchwise self-attention, concatenation is slightly more accurate than star-product and clique-product.
- Controlled experiments: Two linear layers are the default mapping design: pairwise models achieve their highest accuracy with two layers, while patchwise settings have similar accuracy and lower dimensionality costs.A single patchwise mapping layer incurs significant memory and computation costs; bottlenecks reduce both.
- Controlled experiments: Three distinct learnable transformations are generally best, while tied transformations achieve comparable accuracy at higher FLOP counts.Distinct β enables different bottleneck reduction factors r1 and r2, which can lower FLOP consumption.
- Robustness and generalization: Pairwise self-attention is less vulnerable than convolutional or patchwise networks to rotated images and adversarial attacks, although all models suffer from domain shift.With four attack iterations, pairwise and patchwise SAN accuracy is 21.8% and 24.8%, versus 11.8% for ResNet50.
6. Conclusion
The paper finds that fully self-attention-based image recognition models can match or exceed convolutional baselines, with patchwise and vector attention especially strong. Experiments also report substantially greater robustness for self-attention models under adversarial attacks.
- Self-attention models are evaluated in pairwise and patchwise forms, with vector attention introduced for both.The vector formulation efficiently adapts weights across spatial dimensions and channels.
- Self-attention models are much more robust than convolutional networks to adversarial attacks on the val-original set.Attack success rate is lower-is-better, while top-1 accuracy under attack is higher-is-better.
- Pairwise self-attention networks match or outperform convolutional baselines, while patchwise models substantially outperform them.These findings indicate that comparable or higher discriminative power is possible without relying exclusively on convolution.
- Vector self-attention substantially outperforms scalar dot-product attention.