Source-linked AI summary
Are Transformers More Robust Than CNNs?
Yutong Bai, Jieru Mei, Alan Yuille, Cihang Xie
TL;DR
Prior claims that Transformers are more robust than CNNs were based on unfair comparisons involving model scale and training frameworks. This paper aligns those factors and finds that CNNs can match Transformers against adversarial attacks, whereas Transformers retain stronger out-of-distribution generalization associated with self-attention-like architecture.
Problem
Prior Transformer–CNN robustness comparisons used different model scales and training frameworks, leaving their claimed robustness advantage insufficiently established.
Method
The paper fairly compares similarly capable DeiT-S and ResNet-50 under a unified training setup using adversarial and out-of-distribution robustness evaluations.
Results
Transformers are no more robust than CNNs against adversarial attacks when CNNs adopt Transformer training recipes, but Transformers substantially outperform CNNs on out-of-distribution samples without external large-scale pre-training.
Takeaways & Limitations
Adversarial robustness can be matched through training recipes, while stronger out-of-distribution generalization is largely associated with Transformers’ self-attention-like architectures.
Abstract
from arXiv · showhide
Transformer emerges as a powerful tool for visual recognition. In addition to demonstrating competitive performance on a broad range of visual benchmarks, recent works also argue that Transformers are much more robust than Convolutions Neural Networks (CNNs). Nonetheless, surprisingly, we find these conclusions are drawn from unfair experimental settings, where Transformers and CNNs are compared at different scales and are applied with distinct training frameworks. In this paper, we aim to provide the first fair & in-depth comparisons between Transformers and CNNs, focusing on robustness evaluations. With our unified training setup, we first challenge the previous belief that Transformers outshine CNNs when measuring adversarial robustness. More surprisingly, we find CNNs can easily be as robust as Transformers on defending against adversarial attacks, if they properly adopt Transformers' training recipes. While regarding generalization on out-of-distribution samples, we show pre-training on (external) large-scale datasets is not a fundamental request for enabling Transformers to achieve better performance than CNNs. Moreover, our ablations suggest such stronger generalization is largely benefited by the Transformer's self-attention-like architectures per se, rather than by other training setups. We hope this work can help the community better understand and benchmark the robustness of Transformers and CNNs. The code and models are publicly available at https://github.com/ytongbai/ViTs-vs-CNNs.
1 Introduction
The paper argues that prior claims of Transformer superiority in robustness rely on mismatched model scales and training frameworks. Using a unified setup, it finds CNNs can match Transformers on adversarial robustness, while Transformers retain an advantage on out-of-distribution generalization linked to self-attention-like architecture.
- Motivation: Prior robustness comparisons used unequal model scales and distinct training frameworks, making claims that Transformers outperform CNNs difficult to establish.The mismatches include model capacity, training datasets, epochs, and augmentation strategies.
- Approach: The benchmark fairly compares similarly capable DeiT-S and ResNet-50 under a unified setup across adversarial and out-of-distribution robustness.DeiT-S has approximately 22 million parameters and ResNet-50 approximately 25 million.
- Findings: Under unified training, CNNs can attain similar robustness to Transformers against both perturbation-based and patch-based adversarial attacks when adopting Transformer training recipes.This reverses conclusions from earlier comparisons that favored Transformers.
- Findings: Transformers substantially outperform CNNs on out-of-distribution samples even without external large-scale pre-training.The evaluation includes ImageNet-A, ImageNet-C, and Stylized-ImageNet.
- Findings: Ablations suggest self-attention-like architecture, rather than other training setups, is the main contributor to stronger out-of-distribution generalization.Other training adjustments produce only subtle effects in this setting.
2 Related Works
The related work contrasts CNNs’ established role with Transformers’ emergence in vision and reviews robustness evaluations for distribution shift and adversarial attacks. It also motivates fairer comparisons by identifying scale and training discrepancies in prior studies.
- Visual architectures: Vision Transformers initially achieved strong results especially when pre-trained on extremely large datasets, while later work improved their training pipelines and architectures.DeiT demonstrated ImageNet training without external data.
- Robustness evaluations: Out-of-distribution benchmarks evaluate performance when test data differ from training data, including ImageNet-C, Stylized-ImageNet, and ImageNet-A.These benchmarks address the limited ability of finite training datasets to cover real-world distributions.
- Comparison methodology: Prior Transformer–CNN robustness comparisons differed in model capacity or training frameworks, motivating aligned comparisons of architecture and training setup.The paper identifies these inconsistencies as affecting the strength of previous conclusions.
3 Settings
The study uses ResNet-50 and DeiT-S with aligned 100-epoch training and evaluates two robustness axes: adversarial attacks and out-of-distribution generalization. Its attack suite includes PGD, AutoAttack, and Texture Patch Attack, while distribution-shift tests use three ImageNet benchmarks.
- Training CNNs: ResNet-50 is the default CNN, trained for 100 epochs with momentum-SGD and scheduled 10× learning-rate reductions.The initial learning rate is 0.1, with reductions at epochs 30, 60, and 90.
- Training Transformers: DeiT-S is the default Transformer, trained with the DeiT recipe but limited to the same 100-epoch schedule as ResNet-50.The setup removes Erasing, Stochastic Depth, and Repeated Augmentation to avoid over-regularization.
- Evaluation axes: The robustness evaluation covers adversarial examples and out-of-distribution samples as two complementary testing axes.Adversarial examples use imperceptible perturbations or small patches that can induce incorrect predictions.
- Adversarial evaluation: The adversarial suite includes PGD, parameter-free AutoAttack, and Texture Patch Attack using predefined texture patches.AutoAttack combines PGD variants, FAB, and Square Attack.
- Out-of-distribution evaluation: The out-of-distribution suite contains ImageNet-A, ImageNet-C, and Stylized-ImageNet.ImageNet-C measures robustness to 75 common image corruptions, while ImageNet-A uses challenging real-world recognition scenarios.
4 Adversarial Robustness
The study evaluates adversarial robustness under perturbation- and patch-based attacks, showing that training choices strongly affect comparisons between ResNet-50 and DeiT-S. With suitable augmentation warm-up or activation changes, CNN robustness can match or exceed Transformer robustness.
- Perturbation-Based Attacks: At perturbation radius ϵ = 0.001, DeiT-S exceeds ResNet-50 on AutoAttack robustness, 22.1% vs. 17.8%.At the standard radius 4/255, both models achieve 0% robustness without adversarial training.
- Adversarial Training: Adversarial training uses a min-max objective with on-the-fly adversarial examples and PGD-1 constrained to ϵ = 4/255.The training samples optimize loss over allowed perturbations while reducing training cost through single-step PGD.
- Adversarial Training: Combining strong augmentation with adversarial attacks initially over-regularizes DeiT-S, reducing its PGD-5 robustness to approximately 4%.RandAug, Mixup, and CutMix make the resulting training samples too difficult for DeiT-S to learn.
- Adversarial Training: A ten-epoch augmentation warm-up raises DeiT-S PGD-5 robustness from approximately 4% to approximately 44%.The strategy progressively increases augmentation strength during epochs 0–9.
- Adversarial Training: Replacing ResNet-50’s ReLU with GELU improves PGD-100 robustness by 8.01% and matches DeiT-S on PGD-100 and AutoAttack.ResNet-50 + GELU reaches 40.27% versus DeiT-S’s 40.32% on PGD-100, and 35.51% versus 35.50% on AutoAttack.
- Patch-Based Attacks: CutMix improves ResNet-50’s TPA robustness by 29.4%, while CutMix plus RandAug reaches 52.4%, exceeding DeiT-S’s 47.7%.With ten attacking patches, ResNet-50 reaches 34.5% TPA robustness, outperforming DeiT-S by 5.6%.
5 Robustness on Out-of-distribution Samples
Across out-of-distribution robustness benchmarks, DeiT-S consistently outperforms ResNet-50, even after aligning CNN training recipes and comparing models with similar clean accuracy. Ablations indicate that Transformer self-attention-like architecture, rather than training setup alone, largely underlies this advantage.
- Direct comparison: DeiT-S exceeds ResNet-50 by +9.0% on ImageNet-A, +9.9 on ImageNet-C and +4.7% on Stylized-ImageNet under default training.These comparisons use ImageNet-A, ImageNet-C and Stylized-ImageNet to assess different aspects of out-of-distribution robustness.
- Aligning augmentation strategies: RandAug + Mixup improves vanilla ResNet-50 by 3.0% on ImageNet-A, 4.6 on ImageNet-C and 2.4% on Stylized-ImageNet.This is the strongest augmentation combination found in the ResNet-50 ablation.
- Best ResNet recipes: DeiT-S still surpasses ResNet-50-Best by +5.9% on ImageNet-A, +4.3 on ImageNet-C and +2.2% on Stylized-ImageNet.ResNet-50-Best uses M-SGD, cosine decay, RandAug and Mixup, so recipe differences do not explain the full gap.
- Distillation: Switching distillation roles gives DeiT-S improvements of 6.4% on ImageNet-A, 6.3 on ImageNet-C and 3.7% on Stylized-ImageNet over ResNet-50.The result suggests DeiT's robustness generalization is rooted in Transformer architecture and is not transferred to ResNet through distillation.
- Hybrid architecture: Hybrid-DeiT improves over ResNet-50 by +1.1% on ImageNet-A and +2.5% on Stylized-ImageNet, while remaining below pure DeiT-S.The authors identify Transformer self-attention-like architecture as essential for boosting out-of-distribution performance.
- 300-epoch training: Under 300-epoch training, DeiT-S still outperforms ResNet-50 by 10.4% on ImageNet-A, 7.5 on ImageNet-C and 5.6 on Stylized-ImageNet.The paper reports the same qualitative conclusion as in the 100-epoch setup.
6 Conclusion
The paper argues that fair comparisons overturn the claim that Transformers are universally more robust than CNNs. Under unified training, CNNs match Transformers on adversarial robustness, while self-attention-like architectures remain key for out-of-distribution generalization.
- Prior Transformer–CNN robustness comparisons were unfair because they differed in training datasets, model scales, and training strategies.
- Transformers are no more robust than CNNs against adversarial attacks under the paper’s unified training setup.
- CNNs can achieve similar robustness to Transformers against both perturbation-based and patch-based adversarial attacks by adopting Transformer training recipes.
- For out-of-distribution generalization on ImageNet-A, ImageNet-C, and Stylized ImageNet, self-attention-like architectures are identified as the key factor.
- The study aims to improve understanding and fair benchmarking of robustness in Transformers and CNNs.