PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers are Robust Learners

Sayak Paul, Pin-Yu Chen

arXiv:2105.07581v3cs.CVcs.LG

TL;DR

The paper addresses the limited understanding of ViT robustness and its underlying causes. It compares ViTs with BiT across six ImageNet robustness datasets and uses six experiments to analyze the observed differences, finding consistently stronger robustness for ViTs.

  • Problem

    Robustness evaluation and attribution for Vision Transformers remain largely unexplored despite their strong standard-accuracy results.

  • Method

    The study compares ViT models with BiT using six diverse ImageNet datasets and six experiments analyzing masking, Fourier sensitivity, energy distribution, and loss landscapes.

  • Results

    ViTs significantly outperform comparable BiTs across the evaluated robustness datasets and show stronger robustness under common corruptions and perturbations.

  • Takeaways & Limitations

    The analyses identify image masking behavior, low Fourier-spectrum sensitivity, wider energy distribution, and smoother loss landscapes as properties associated with ViT robustness.

  • Takeaways & Limitations

    The study does not investigate how large-scale pre-training and attention-module aspects relate to robustness.

Abstract

from arXiv · show

Transformers, composed of multiple self-attention layers, hold strong promises toward a generic learning primitive applicable to different data modalities, including the recent breakthroughs in computer vision achieving state-of-the-art (SOTA) standard accuracy. What remains largely unexplored is their robustness evaluation and attribution. In this work, we study the robustness of the Vision Transformer (ViT) against common corruptions and perturbations, distribution shifts, and natural adversarial examples. We use six different diverse ImageNet datasets concerning robust classification to conduct a comprehensive performance comparison of ViT models and SOTA convolutional neural networks (CNNs), Big-Transfer. Through a series of six systematically designed experiments, we then present analyses that provide both quantitative and qualitative indications to explain why ViTs are indeed more robust learners. For example, with fewer parameters and similar dataset and pre-training combinations, ViT gives a top-1 accuracy of 28.10% on ImageNet-A which is 4.3x higher than a comparable variant of BiT. Our analyses on image masking, Fourier spectrum sensitivity, and spread on discrete cosine energy spectrum reveal intriguing properties of ViT attributing to improved robustness. Code for reproducing our experiments is available at https://git.io/J3VO0.

1 Introduction

Transformers reduce reliance on hand-designed inductive biases and have become competitive with SOTA CNNs in vision. This work asks whether ViTs are inherently more robust and uses systematic evidence to answer yes and investigate why.

  • Transformers reduce inductive biases through self-attention, which quantifies pairwise entity interactions within input data.These properties support learning hierarchies and alignments across data modalities.
  • By 2020, vision Transformers reached parity with SOTA CNNs on standard image-recognition tasks.
  • This work evaluates whether ViTs are more robust than SOTA CNNs and provides empirical analyses of the reasons behind any improvement.The analyses target semantic shifts, common corruptions and perturbations, and natural adversarial examples.
  • Prior robustness studies of ViTs had relatively limited scope.
  • The study uses six diverse ImageNet datasets and six experiments covering masking, energy or loss landscapes, and sensitivity to high-frequency artifacts.

2 Related Work

Related work established Transformers for image generation, pre-training, and classification, while subsequent studies began examining ViT robustness. These studies reported robustness advantages but differed in scope and proposed explanations.

  • Image-GPT applied Transformers to unsupervised pixel-level image pre-training, while ViT minimally modified the original architecture for image classification.ViT’s reduced inductive biases increase its need for relatively larger pre-training datasets.
  • DeiT achieved 85.2% top-1 accuracy on ImageNet-1k without external data using distillation from a RegNet teacher.
  • Concurrent robustness studies found ViTs outperform CNNs against adversarial perturbations, attributing gains to high-frequency features or reduced convolutional interference.The cited studies examined different robustness perspectives from the present work.

3 Robustness Performance Comparison on ImageNet Datasets

The paper compares ViT and BiT across six ImageNet robustness benchmarks spanning corruptions, perturbations, semantic shifts, natural adversarial examples, and out-of-distribution detection. ViT generally shows better robustness than BiT under comparable parameter and pre-training regimes, while the paper examines pre-training and self-attention as possible explanations.

  • Comparison Scope: The study controls for shared pre-training strategies while noting differences in dropout, optimizer choice, and other training details between ViT and BiT.The analysis focuses on publicly available BiT and ViT models and does not investigate how later regularization techniques relate to robustness.
  • Evaluation Setup: ViT and BiT are compared using six ImageNet robustness datasets covering common corruptions, semantic shifts, natural adversarial examples, and out-of-distribution detection.The comparison also includes ResNet50V2 as a baseline.
  • Overall Comparison: Across the studied datasets, ViT exhibits significantly better robustness than BiT models with comparable parameter counts.The paper presents Section 4 as an attribution analysis of this robustness difference.
  • Common Corruptions: ViT variants consistently outperform BiT across ImageNet-C corruptions except contrast, despite differing parameter-count regimes across some model pairings.ImageNet-C evaluation uses all 19 corruptions at severity level 5 and reports mean top-1 accuracy.
  • Common Perturbations: ViT robustness is significantly better than BiT and AugMix on ImageNet-P common perturbations, evaluated with mean flip rate and mean top-5 distance.Both metrics are lower-is-better robustness measures.
  • Distribution Shifts and Natural Adversarial Examples: ViT performs better than BiT under ImageNet-R semantic shifts and achieves 4.3x higher top-1 accuracy than BiT-m r101x3 on ImageNet-A.ImageNet-R tests renditions across domains, while ImageNet-A contains natural images that induce misclassification.

4 Why ViT has Improved Robustness?

The experiments attribute ViT’s improved robustness to attention’s global contextual processing, stronger pretraining, and sensitivity patterns spanning image frequencies.

  • Six systematically designed experiments examine sources of ViT robustness using qualitative and quantitative analyses.
  • Attention is Crucial for Improved Robustness: Attention-fused models are evaluated to test whether contextual dependencies contribute to ViT’s performance on three robustness benchmarks.
  • ViT consistently outperforms similar BiT models across six robustness benchmark datasets, including corruptions, semantic shifts, natural adversarial examples, and out-of-distribution detection.
  • ViT Has Better Robustness to Image Masking: Random masking experiments show ViT consistently beats BiT, including when square portions of input images are randomly masked.
  • Fourier Spectrum Sensitivity: ViT and BiT remain robust to most Fourier-frequency regions, while ResNet50V2 loses consistency in high-frequency regions.
  • Discrete Cosine Energy Spectrum: Attacking ViT requires affecting almost the entire frequency spectrum, whereas perturbations for BiT and ResNet-50 concentrate more around low frequencies.
  • ViT Has Smoother Loss Landscape to Input Perturbations: Under attack steps, ViT’s cross-entropy loss increases more slowly than BiT’s, supporting a smoother loss landscape to input perturbations.

5 Conclusion

The work evaluates ViT robustness across six ImageNet datasets and concludes that ViT outperforms CNN baselines while exhibiting several properties associated with robustness.

  • ViT significantly outperforms BiT and ResNet50V2 across six ImageNet robustness datasets.The evaluation covers common corruptions, semantic shifts, natural adversarial examples, and out-of-distribution detection.
  • ViT’s robustness analyses examine large-scale pre-training, attention, random masking, Fourier-spectrum sensitivity, energy distribution, and adversarial loss landscapes.
  • Self-attention computes query, key, and value projections and returns outputs weighted by pairwise attention scores.Multiple attention heads are stacked within Transformer blocks alongside layer normalization, feed-forward networks, and skip connections.
  • ViT prepends a class token to encoded image patches, and this token becomes the final image representation passed to the task head.
  • ViT divides 224 × 224 images into 196 patches of size 16 × 16, flattening each RGB patch into 768 dimensions.A linear transformation is used for patch encoding, while positional information is added afterward.

C Additional Results on Attention

Attention visualizations indicate that ViT often uses broader image context than BiT, including under difficult lighting and corruption conditions, although some attention can be spurious.

  • ViT maintains correct predictions under dark lighting, including low-confidence cases where its visual focus is not clearly localized.The study uses Attention Rollout and visualizes both high-confidence and low-confidence correct predictions.
  • The Grad-CAM comparison computes target-class gradients from BiT’s last convolutional block and adapts the analysis for ViT’s architecture.
  • ViT uses information from different image regions, whereas BiT’s Grad-CAM explanations are more local and central.The authors connect ViT’s broader context use to robustness when attentive regions are masked.
  • ViT’s attention can include spurious regions that are not explanatory, suggesting an area for future research.

D Additional Experiments for Pre-training

The pre-training experiments examine ImageNet-A, where larger datasets, longer schedules, and greater model capacity are associated with improved performance across model variants.

  • ImageNet-A is used because its images often contain noncentral objects and multiple objects, making robust recognition more challenging.
  • Longer pre-training schedules combined with larger datasets can improve performance on ImageNet-A.
  • Performance on ImageNet-A also improves as model capacity increases.
  • Table 11 compares BiT-m and ViT variants, including parameter-comparable BiT m-r101x3, ViT L-16, and ViT L-32 models.

F Additional Results on ImageNet-C

Additional ImageNet-C analyses show that ViT generally retains robust predictions under corruptions, though contrast corruption is a notable weakness and BiT predictions can flip under slight corruption.

  • ViT performs particularly poorly on ImageNet-C’s contrast corruption at severity level 5.The individual-corruption analysis covers 15 corruption types.
  • The ImageNet-C comparison reports unnormalized corruption errors for BiT m-r101x3 and ViT L-16 across 15 corruptions.
  • ViT uses global context to retain strong-confidence predictions under corruptions that cause BiT’s predictions to flip.The Grad-CAM examples sample images across different corruption severity levels.
  • Figure 13 compares cases where BiT is wrong while ViT remains correct, using Grad-CAM visualizations and prediction confidence scores.

G Additional Results on ImageNet-P

ImageNet-P perturbation results are reported using unnormalized flip rates and top-5 distances for each perturbation type.

  • Flip rates and top-5 distances are reported separately for each ImageNet-P perturbation type.The scores are unnormalized and are not scaled using corresponding AlexNet scores.

H Random Masking with Cutout

The appendix visualizes how BiT and ViT predictions change as Cutout masking increases, while cautioning that these plots are not conclusive.

  • Figure 14 compares prediction changes for BiT m-r101x3 and ViT L-16 across Cutout masking factors.The first row shows BiT-m r101x3 and the second row shows ViT L-16.
  • The masking visualizations are provided for context and should not be treated as conclusive evidence.

I Magnitude Spectrum and High-Frequency Components

This appendix provides visualizations related to Fourier analysis, including frequency-domain magnitude spectra and raw high-frequency image components, alongside corruption tables.

  • Figure 15 visualizes natural-image magnitude spectra in the frequency domain and raw high-frequency components.These visualizations support the paper’s Fourier analysis in Section 4.4.

J Adversarial Perturbations

The appendix examines adversarial perturbations and PGD loss landscapes for BiT and ViT, complementing per-perturbation ImageNet-P results.

  • ViT L-16 generally produces smoother adversarial perturbations than BiT-m r101x3 under the described PGD visualization setup.The attack uses Adam with a learning rate of 1e-3.
  • Figure 17 shows PGD loss plots for five individual, non-cherry-picked ImageNet-1k validation images.The examples are intended to isolate the results shown in Figure 8.
Loading 2105.07581v3…