Source-linked AI summary
Do Vision Transformers See Like Convolutional Neural Networks?
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy
TL;DR
The paper asks whether Vision Transformers solve image-classification tasks like CNNs or learn different visual representations. It compares their internal representations with CKA and related analyses, finding clear architectural differences associated with early global information and strong feature propagation in ViTs. These findings also identify the importance of local information, dataset scale, and classification methods for spatial and transferable representations.
Problem
It is unclear whether Vision Transformers learn the same visual representations and inductive biases as CNNs despite comparable or superior image-classification performance.
Method
The paper compares ViT and CNN layer representations using CKA, attention-distance analyses, skip-connection analyses, spatial-localization tests, and linear probes across training and transfer settings.
Results
ViTs differ clearly from CNNs, with early global-information aggregation, strong lower-to-higher feature propagation, preserved spatial information, and representation changes linked to dataset scale and classification methods.
Takeaways & Limitations
ViTs are not simply reproducing CNN representations: their global attention and residual connections shape distinctive internal structures, while local early information remains important for strong performance.
Takeaways & Limitations
CKA compresses representation comparisons into a single scalar, so finer-grained methods may reveal additional insights and variation.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.
1 Introduction
Vision Transformers achieve comparable or superior image-classification performance to CNNs, prompting questions about whether they learn similar or novel visual representations. The paper investigates these differences, their origins, and consequences for classification and transfer learning.
- Vision Transformers can match or exceed CNN performance on large-scale image classification tasks.
- The paper asks whether ViTs reproduce convolutional inductive biases or develop different task representations.
- ViTs show more uniform representations across layers, with greater similarity between lower and higher layers than CNNs.
- The study examines local and global information, skip connections, spatial localization, and dataset scale as factors affecting representations and transfer learning.
2 Related Work
Prior vision-Transformer research explored local attention, CNN–attention hybrids, and applications to smaller images. This paper focuses on the minimally modified Vision Transformer to compare its representational structure with CNNs using representation-similarity methods and linear probes.
- Earlier studies adapted attention to local receptive fields, combined CNNs with self-attention, or applied Transformers to smaller images.
- The Vision Transformer makes fewer architectural modifications than these approaches, making it a useful subject for comparison with CNNs.
- The paper uses neural-network representation-similarity techniques to compare internal representations within and across vision architectures.
- Linear probes are also applied in the study of model representations.
3 Background and Experimental Setup
The study compares representative ResNet and ViT models trained primarily on JFT-300M, with additional ImageNet and transfer-learning settings. CKA provides quantitative comparisons of layer representations using Gram matrices and normalized centered similarity.
- The experiments compare ResNet50x1, ResNet152x2, ViT-B/32, ViT-B/16, ViT-L/16, and ViT-H/14 models.Unless otherwise specified, models are trained on JFT-300M; ImageNet and transfer-learning benchmarks are also investigated.
- Centered kernel alignment compares representations from two layers evaluated on the same examples, even when their neuron counts differ.It operates on activation matrices and their Gram matrices.
- CKA compares layer-pair similarity structures through Gram matrices derived as K = XXT and L = YYT.
- The study visualizes within-model and cross-model layer similarities with CKA heatmaps whose axes index layers from input to output.
- The HSIC-based CKA computation uses centered Gram matrices and normalization to support meaningful representation comparisons.The large-scale analysis approximates the unbiased HSIC estimator with minibatches.
4 Representation Structure of ViTs and Convolutional Networks
ViTs and ResNets exhibit distinct layer-representation structures: ViT layers are more uniformly similar, whereas ResNets form clearer stages. Cross-model comparisons show that ResNet depth maps unevenly onto ViT depth, with the highest ViT layers remaining distinct.
- ViT representations have relatively uniform similarity across layers, while ResNets show staged structures with weaker lower-to-higher-layer similarity.The comparison uses CKA similarities between pairs of layers within each model.
- The lower half of ResNet layers is similar to approximately the lowest quarter of ViT layers.
- The top half of ResNet layers is approximately similar to the next third of ViT layers.
- The final third of ViT layers is less similar to all ResNet layers, likely because it mainly manipulates the CLS token representation.
- Together, the results indicate that ViT lower layers differ from ResNet lower layers, while ViTs more strongly propagate representations between lower and higher layers.
5 Local and Global Information in Layer Representations
ViTs aggregate more global information in lower layers than ResNets, producing quantitatively different features. Large-scale pretraining helps early attention layers learn local information that supports strong performance.
- Effect of scale: ImageNet-only training prevents lower ViT attention layers from learning local attention and yields much worse performance in large ViT models.The result suggests local information early in the network remains important for strong performance.
- Representation differences: Lower ResNet representations are most similar to ViT features from locally attending heads, with similarity decreasing as attention becomes more global.The comparison uses CKA between subsets of first-block ViT heads and lower ResNet layers.
- Global information: Early ViT self-attention aggregates more global information than CNN receptive fields, creating different lower-layer representations.Self-attention uses attention distance to quantify local versus global information, while CNNs use fixed receptive fields.
- Effective receptive fields: ViT effective receptive fields start larger than ResNet fields and become substantially more global midway through the network.ResNet receptive fields grow gradually, whereas ViT fields shift from local to global.
6 Representation Propagation through Skip Connections
ViT residual connections strongly propagate representations across layers, helping produce a uniform similarity structure. Their influence shifts from preserving the CLS token early to preserving spatial tokens later.
- Skip-connection propagation: ViT skip connections are much more influential than ResNet skip connections and exhibit a phase transition from CLS-token to spatial-token propagation.The phase transition reverses which token representations are primarily carried by the skip branch across the network.
- Measurement: The analyzed skip connections add identity outputs after both self-attention and MLP transformations, with their influence measured by a norm ratio.The ratio compares the hidden representation from the skip branch with the transformed long branch.
- Token-specific propagation: In the first half of ViT, the CLS token is primarily propagated through the skip branch, while spatial tokens receive larger long-branch contributions.In the second half, this pattern reverses.
- Intervention: Removing a middle-block skip connection partitions representations across that block and causes a 4% performance drop.The intervention demonstrates that skip connections help maintain ViT’s uniform representation structure.
7 Spatial Information and Localization
Higher-layer ViT tokens preserve input spatial information more faithfully than ResNet representations, but classification design affects localization. Global average pooling reduces localization and makes higher-layer tokens more globally similar.
- Spatial localization: Higher-layer ViT tokens are most similar to corresponding input patches, whereas ResNet representations show significantly weaker localization.Edge ViT tokens also resemble edge patches, while interior tokens are especially well localized.
- Classification method: Training ViTs with global average pooling reduces higher-layer localization compared with using a classification token.The experiment directly tests whether the classification method contributes to the architecture difference.
- Evaluation: Linear probes evaluate token representations across layers using 10-shot ImageNet classification and compare separate-token and aggregation strategies.The probes use closed-form few-shot linear regression across different model layers.
- Linear probes: ViT-GAP reaches comparable highest-layer accuracy with a single representation to that obtained by pooling all tokens, indicating globally similar higher-layer tokens.This result complements the weaker localization observed for GAP models.
8 Effects of Scale on Transfer Learning
Pretraining dataset scale affects ViT representations unevenly: lower layers stabilize with relatively little data, while higher layers require more data, especially in larger models. Larger pretraining datasets also improve intermediate representations for transfer learning.
- Representation similarity: Even with 3% of JFT-300M, lower-layer representations remain similar to full-data models, while higher layers require substantially more data.The dependence on dataset scale is especially strong for larger ViT models.
- Transfer learning: JFT-300M-pretrained larger ViTs achieve a 30% absolute accuracy gap over ImageNet-only models using middle-layer representations.The comparison uses linear probes on ImageNet classes for ViT-L/16 and ViT-H/14 models.
- Architecture comparison: Among JFT-300M-pretrained models, larger ViTs learn stronger intermediate representations than ResNets under linear-probe evaluation.The comparison evaluates representations from different layers on ImageNet.
9 Discussion
The study’s interpretation is bounded by its reliance on CKA as a scalar similarity measure, despite complementary interventions and linear-probe analyses. Its broader discussion situates the results within varied models, datasets, and evaluation procedures.
- Limitations: CKA compresses representation similarity into a single scalar, so finer-grained methods may reveal additional structure or variation.The authors complement CKA with interventional tests and linear probes, but identify finer-grained analysis as an open limitation.
- Experimental scope: The study compares representative ViT and CNN models across JFT-300M, ImageNet, and transfer-learning benchmarks.The model set includes ViT-B/32, ViT-L/16, ViT-H/14, ResNet50x1, and ResNet152x2, with dataset and benchmark variation.
- Methods: The supplemental material provides additional experimental details for CKA estimation, linear-probe training, datasets, and pooling procedures.These details specify minibatch sampling, regularized least-squares probes, dataset splits, and representation pooling.
- Methods: Analyses use CKA, linear probes, and controlled representation comparisons to examine internal structure and transfer behavior.CKA is computed over layer pairs and intermediate representations, while probes evaluate information in tokens and layers.
B Additional Representation Structure Results
Additional CKA results reinforce that ViTs have a more uniform representation structure across layers, while many early ResNet layers correspond to fewer early ViT layers.
- Representation similarity: ViT representations show a more uniform similarity structure across layers, with strong similarity between lower and higher layers.The additional heatmaps reproduce the main representation-structure conclusion.
- Cross-model comparison: Many early ResNet layers are similar to a smaller number of ViT layers in cross-model representation comparisons.The supplemental comparison reports this pattern across additional model pairs.
C Additional Local/Global Information Results
The additional receptive-field and localization analyses show that ViT attention integrates global information while residual connections preserve spatially localized representations. This behavior varies with layer, model scale, and classification design.
- Receptive fields: ViT attention sublayers have globally distributed pre-residual receptive fields, but strong skip connections keep post-residual representations tied to input patch locations.The contrast is explicit in ViT-B/32: attention integrates across the image, while residual pathways preserve localization.
- Local and global information: Lower ViT attention layers attend both locally and globally, whereas higher layers show different attention-distance patterns.Additional attention-distance plots examine ViT-H/14 and compare models trained under different data scales.
- Localization: ViT-H/14 and higher layers of ViT-L/16 and ViT-B/32 remain well localized, with spatially corresponding tokens most similar to corresponding image patches.Edge tokens show broader similarity across edge positions, while interior tokens retain clearer positional correspondence.
- Localization: ResNet layers below the final layer localize better than the final layer, but less clearly than CLS-trained ViT models.The comparison concerns localization heatmaps for intermediate ResNet layers and ViT representations.
- Probe evaluation: Linear probes evaluate spatial information by measuring per-token accuracy across selected layers in CLS-trained ViT, GAP-trained ViT, and ResNet50 models.The probe plots use average test accuracy and have different value scales across subplots.
- Representation propagation: Spatial-token representations change more in early ViT layers, while later blocks remain close to their skip-connection representations.The CLS token shows the opposite pattern, changing rapidly in later layers, mostly through MLP blocks.
F Additional results on linear probes
Additional linear-probe results clarify how classification strategy shapes token information. Native CLS and GAP evaluations perform similarly, while token-level information and transfer stability vary across layers, datasets, and architectures.
- Additional experiments: Additional experiments extend the linear-probe comparisons to CIFAR-10 and CIFAR-100 and add ViT and ResNet model variants.The supplemental datasets use 45,000 training images and 5,000 validation images from each training set.
- Aggregation methods: CLS and GAP models perform very similarly when evaluated with their native representation aggregation methods.The native methods are first-token evaluation for CLS and GAP evaluation for GAP.
- Aggregation methods: For CLS models, pooled representations match first-token evaluation through most layers before performance plateaus near the end.This pattern indicates that the CLS token contributes importantly to late-layer information aggregation.
- Aggregation methods: For GAP-trained models, first-token probes improve substantially in later layers and nearly match standard GAP evaluation at the final layer.The result suggests that tokens become largely interchangeable late in GAP models.
- Per-token probes: Individual-token probes perform poorly throughout ViT-CLS representations, whereas ResNet probes improve toward later layers similarly to ViT-GAP.The contrast reflects the different roles of CLS and GAP-style aggregation in the compared models.
- Finetuning and transfer: Finetuning changes representations more for medical and satellite data than for ImageNet or CIFAR-100, while the first four to five layers remain relatively stable.The pretrained model is from JFT-300M, and the magnitude of change depends on dataset similarity to natural images.
- MLP-Mixer comparison: MLP-Mixer representations form distinct blocks across model sizes and behave more similarly to ViT than to ResNet.The supplemental comparisons report this structural resemblance independently of model size.