PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Vision Transformers by Revisiting High-frequency Components

Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, Wei Liu

arXiv:2204.00993v3cs.CV

TL;DR

ViT models are harder to train than CNNs and require larger training sets, and the paper hypothesizes that weaker high-frequency capture contributes to this gap. It analyzes models and existing techniques by frequency, then proposes HAT, which augments high-frequency components through adversarial training. HAT consistently improves diverse ViT models, including a reported 1.2% top-1 accuracy boost for ViT-B on ImageNet validation.

  • Problem

    ViT models are more difficult to train and require significantly more training samples than CNN models; the paper hypothesizes that weaker high-frequency capture may help explain this.

    Source: §1 · pp. 1–2

  • Method

    The paper analyzes ViT performance on low- and high-frequency components and proposes HAT, adversarial training that augments high-frequency image components.

    Source: §1 · pp. 2–3

  • Results

    HAT boosts top-1 accuracy on the ImageNet validation set by 1.2% for ViT-B and consistently improves diverse ViT models and benchmarks.

    Source: §5.2 · p. 10, §6 · p. 14

  • Takeaways & Limitations

    HAT adds no inference-stage complexity and does not alter the model architecture.

    Source: §1 · p. 3

    HAT increases training time compared with normal training; improving its efficiency is left for future work.

    Source: §6 · p. 14

Abstract

from arXiv · show

The transformer models have shown promising effectiveness in dealing with various vision tasks. However, compared with training Convolutional Neural Network (CNN) models, training Vision Transformer (ViT) models is more difficult and relies on the large-scale training set. To explain this observation we make a hypothesis that \textit{ViT models are less effective in capturing the high-frequency components of images than CNN models}, and verify it by a frequency analysis. Inspired by this finding, we first investigate the effects of existing techniques for improving ViT models from a new frequency perspective, and find that the success of some techniques (e.g., RandAugment) can be attributed to the better usage of the high-frequency components. Then, to compensate for this insufficient ability of ViT models, we propose HAT, which directly augments high-frequency components of images via adversarial training. We show that HAT can consistently boost the performance of various ViT models (e.g., +1.2% for ViT-B, +0.5% for Swin-B), and especially enhance the advanced model VOLO-D5 to 87.3% that only uses ImageNet-1K data, and the superiority can also be maintained on out-of-distribution data and transferred to downstream tasks. The code is available at: https://github.com/jiawangbai/HAT.

1 Introduction

The introduction argues that ViT models are less effective than CNNs at capturing high-frequency image components, a proposed reason for their training difficulty. It examines existing techniques through this frequency perspective and introduces HAT, which augments high-frequency components through adversarial training.

1 Introduction

  • ViT-B outperforms ResNet-50 on low-frequency components but underperforms on high-frequency components, supporting the hypothesis of weaker high-frequency capture.On the original ImageNet validation set, ViT-B has 82.0% top-1 accuracy versus ResNet-50's 79.8%.Source: §1 · p. 2
  • Frequency analysis finds that CNN-teacher distillation, convolutional-like or multi-scale ViT architectures, and RandAugment better exploit high-frequency components than their compared alternatives.RandAugment is reported as more helpful than CutMix and Mixup for capturing high-frequency components.Source: §1 · p. 3
  • HAT augments high-frequency components through adversarial training, jointly training ViT models on clean images and adversarial examples without changing architecture or inference complexity.The paper reports that HAT improves ViT performance by compensating for its ability to capture high-frequency components.Source: §1 · p. 3
  • HAT's results indicate superiority on ImageNet classification and out-of-distribution data, and report benefits from HAT-pretrained models on downstream tasks.Source: §1 · p. 3

2 Related Work

Related work describes ViT's extension to multiple vision tasks and the training strategies developed to improve its performance on smaller datasets. This paper distinguishes its focus as directly compensating for ViT's high-frequency capture ability.

2 Related Work

  • ViT has been extended from image classification to object detection, segmentation, image enhancement, and video processing, alongside ongoing architecture design.Examples include hierarchical shifted-window attention for multi-scale features and outlook attention for finer-level features and contexts.Source: §2 · pp. 3–4
  • Training strategies for ViT on smaller datasets include strong augmentation and regularization, customized augmentations, CNN-teacher distillation, and optimization methods.Examples include CutMix, Mixup, RandAugment, patch diversification, and sharpness-aware minimization.Source: §2 · p. 4
  • Unlike prior strategies described here, this paper focuses on directly compensating for ViT's ability to capture high-frequency image components.Source: §2 · p. 4

3 Revisiting ViT Models from a Frequency Perspective

The section evaluates ViT models through low- and high-frequency image components, finding that their strengths vary by frequency. It also examines how distillation, architecture, and augmentation relate to high-frequency performance.

Frequency Analysis

  • The frequency analysis uses DFT to split images into low- and high-frequency components and evaluates model performance on each across filter sizes.The low-frequency components are shifted to the spectrum center; larger filter sizes retain more low-frequency content in low-pass filtering and more high-frequency content in high-pass filtering.Source: §3 · p. 4

Comparison of ViT and CNN Models

  • ViT-B outperforms ResNet models on low-frequency components but trails them on high-frequency components, despite higher original ImageNet accuracy.The authors suggest this reflects stronger global-context capture but weaker use of local details in ViT-B.Source: §3 · p. 5

Knowledge Distillation and Architectures

  • Advanced ViT architectures perform similarly to ViT-B on low frequencies but better on high frequencies; CNN-teacher distillation also improves high-frequency performance.Distillation raises accuracy from 82.0% to 83.6%; the compared architectures are ConViT-B, Twins-SVT-L, and Swin-B.Source: §3 · p. 5, Figure 2 · p. 5

Data Augmentations

  • CutMix ranks highest on low-frequency components, while RandAugment ranks highest on high-frequency components, with Mixup between them in both comparisons.The authors connect CutMix with global context and suggest RandAugment transformations encourage attention to high-frequency information.Source: §3 · p. 6, Figure 3 · p. 6

4 The Proposed Method

The proposed method augments image high-frequency content with adversarial perturbations during training. In a ViT-B case study, high-frequency perturbations improve ImageNet validation accuracy, while low-frequency perturbations do not.

4.1 Adversarial Training with High-frequency Perturbations

  • HAT directly augments training images with adversarial perturbations and jointly trains ViT models on clean images and altered examples.Its purpose is to address ViT models’ limited ability to capture high-frequency image components.Source: §4.1 · pp. 6–7
  • The objective combines clean-image cross-entropy with a maximized perturbed-image loss and KL divergence, constraining perturbations to the high-frequency domain.The objective weights the perturbed-image cross-entropy and KL terms with hyperparameters α and β.Source: §4.1 · p. 7
  • Adversarial perturbations concentrate more in high frequencies than natural images, motivating HAT’s use of full-frequency perturbations.The paper reports this concentration comparison from averaged Fourier-spectrum heat maps.Source: §4.1 · p. 7, Figure 4 · p. 7
  • The method uses PGD for K steps, accumulates model-weight gradients across steps, and applies adversarial training only in initial epochs before normal training resumes.The ViT-B setting uses 200 adversarial-training epochs.Source: §4.1 · p. 7

4.2 A Case Study using ViT-B

  • 1.0% gain over the baseline: high-frequency adversarial training improves ViT-B on ImageNet validation, while low-frequency perturbations bring no improvement.Full-frequency perturbations also improve ViT-B and are presented as a simpler choice for HAT.Source: Table 1 · p. 8, §4.2 · p. 8

5 Experiments

Experiments evaluate HAT across ImageNet classification, out-of-distribution datasets, downstream tasks, and ablations. HAT improves performance across model variants and settings, with results also indicating benefits under smaller training sets and high-frequency corruptions.

5.1 Experimental Setup

  • Experiments train ViT-T/S/B, Swin-T/S/B, and VOLO-D1/D5 on ImageNet-1K for 300 epochs; the HAT strategy is used only during the first 200 epochs for ViT models.HAT defaults to K=3 and ϵ=2/255; VOLO-D5 instead uses K=2 and ϵ=1/255 to reduce training time.Source: §5.1 · pp. 8–9

5.2 Results on ImageNet Classification

  • +1.2% top-1 accuracy for ViT-B and 87.3% top-1 accuracy for VOLO-D5 at 512×512; HAT with knowledge distillation reaches 84.3%.The comparisons use ImageNet-1K data; Table 3 compares training strategies on ViT-B.Source: Table 2 · p. 9, §5.2 · p. 10, Table 3 · p. 10

5.3 Results on Out-of-distribution Data

  • HAT improves performance across all tested architectures on five out-of-distribution datasets, including ImageNet-C, where lower mCE is better.All models are trained on ImageNet-1K and tested on the five datasets.Source: Table 4 · p. 11, §5.3 · p. 11

5.4 Transfer Learning to Downstream Tasks

  • HAT improves Swin Transformer detection and instance segmentation on COCO and segmentation on ADE20K, with over 1.0% mIOU gains for Swin-T and Swin-S.The reported downstream benefits transfer from ImageNet-pretrained models.Source: §5.4 · pp. 11–12, Table 5 · p. 12, Table 6 · p. 12

5.5 Ablation Studies

  • Medium perturbations (ϵ=2/255 or 3/255) perform best; HAT improves ViT-B across 16K, 32K, and 64K training sets, with +7.1 on the smallest.The training-set sizes are sampled from ImageNet-1K; the authors suggest the gains may be more significant on small sets.Source: §5.5 · pp. 12–13, Table 7 · p. 12, Figure 6 · p. 13

5.6 Discussions

  • ViT-B trained with HAT is less sensitive to additive high-frequency noise than the baseline in Fourier heat maps.The error rates are averaged over ImageNet validation images, with additive noise fixed at ℓ2-norm 15.7.Source: §5.6 · p. 13, Figure 7 · p. 13
  • HAT surpasses normal training at 1.7× and 2.3× computational cost; it also improves ResNet-50 by 0.4% and ViP-Small-7 by 0.6%.The authors report 500 epochs suffice for normal training to converge and describe extension to other models as promising.Source: §5.6 · p. 14, Table 8 · p. 14, Table 9 · p. 14

6 Conclusions and Future Work

The paper concludes that ViT models capture image high-frequency components less effectively than CNNs and proposes HAT as an adversarial-training strategy to address this limitation. It also identifies increased training time as a limitation and points to further work on efficiency and alternative approaches.

6 Conclusions and Future Work

  • The paper finds that ViT models exploit image high-frequency components less effectively than CNNs and presents HAT as a training strategy that improves performance across diverse benchmarks.HAT is based on adversarial training.Source: §6 · p. 14
  • HAT increases training time compared with normal training, motivating future work on improving its efficiency.The paper also proposes exploring other techniques to compensate for ViT models' high-frequency capture ability.Source: §6 · p. 14

A More Related Works

  • Related work spans Fourier-based analyses and methods for CNNs and ViTs, including frequency-domain preprocessing, amplitude mix, and architectures combining attention with convolution.The review also describes theoretical work relating cascaded self-attention to repeated low-pass filtering.Source: A More Related Works · p. 19

B Theoretical Analysis

The analysis states that repeated self-attention diminishes high-frequency information as depth increases. The accompanying algorithm describes HAT's minibatch perturbation and model-weight updates.

B Theoretical Analysis

  • Theorem 1 states that self-attention diminishes high-frequency information as the number of attention applications increases.The theorem is presented for the attention map produced by softmax in ViT models.Source: B Theoretical Analysis · p. 19

C Algorithm Pipeline

  • For each minibatch, the HAT algorithm initializes perturbations at zero and iteratively updates and clips them within the maximum perturbation strength.The perturbation update uses PGD steps and step size η.Source: C Algorithm Pipeline · p. 20
  • The algorithm accumulates model-weight gradients across perturbation steps, then updates the model weights.The gradients are computed alongside perturbation gradients during the inner loop.Source: C Algorithm Pipeline · p. 20

D Implementation Details

This section combines HAT with knowledge distillation by replacing the cross-entropy term with a distillation loss while keeping the optimization and hyperparameters unchanged.

D.1 Combining HAT with Knowledge Distillation

  • HAT's objective combines clean-image distillation loss with a maximized perturbed-image objective containing distillation and KL-divergence terms.The perturbation is constrained by an ℓ∞ bound.Source: §D.1 · p. 20
  • The method replaces the cross-entropy loss in the HAT objective with the distillation loss used in DeiT.This substitution defines the knowledge-distillation variant of HAT.Source: §D.1 · p. 20
  • The distillation loss uses the teacher model's hard decision, and the optimization and hyperparameters remain unchanged.The loss combines cross-entropy terms for the ground-truth label and teacher decision.Source: §D.1 · p. 20

D.2 Object Detection and Instance Segmentation

The section describes evaluation setups for object detection and instance segmentation on COCO and semantic segmentation on ADE20K. It also reports ImageNet validation accuracy for three training and pre-training settings.

D.2 Object Detection and Instance Segmentation

  • Object detection and instance segmentation are evaluated on COCO 2017 using Cascade Mask R-CNN, multi-scale training, and a 36-epoch schedule.The dataset has 118K training, 5K validation, and 20K test-dev images; the setup uses AdamW with initial learning rate 0.0001, weight decay 0.05, and batch size 16.Source: §D.2 · pp. 20–21
  • ImageNet validation top-1 accuracy is 76.7% for vanilla training, 84.5% for ImageNet-22K pre-training, and 83.8% for ImageNet-1K SimMIM pre-training.Fig. 8 compares the methods on low- and high-pass filtered validation sets at different filter sizes.Source: Figure 8 · p. 21

D.3 Semantic Segmentation

  • Semantic segmentation is evaluated on ADE20K with UperNet, following the original Swin Transformer setup and multi-scale inference.ADE20K has 20K training, 2K validation, and 3K test images; training uses 160K iterations across eight GPUs.Source: §D.3 · p. 21

E.1 Large-scale Pre-training

HAT fine-tuning improves ViT-B accuracy under SimMIM pre-training, while analyses report that supervised and self-supervised pre-training help exploit high-frequency components. The distillation comparison favors a CNN teacher over a ViT teacher for ViT-B's high-frequency capture.

E.1 Large-scale Pre-training

  • ImageNet-22K supervised and SimMIM self-supervised pre-training both help exploit high-frequency components, especially SimMIM.Source: §E.1 · p. 21
  • 83.9% top-1 accuracy is achieved by ViT-B with HAT fine-tuning under SimMIM pre-training, versus 83.8% reported in the paper and 83.6% in reproduction without HAT.Adversarial training is used for the first 80 epochs, followed by normal training for 20 epochs.Source: §E.1 · p. 21

E.2 Knowledge Distillation

  • ViT-B reaches 82.0% top-1 accuracy without distillation, 82.8% with a Swin-B teacher, and 83.6% with a RegNetY-16GF teacher.The analysis indicates that the CNN teacher is more helpful than the ViT teacher for capturing high-frequency components.Source: Figure 9 · p. 22, §E.2 · p. 22
Loading 2204.00993v3…