PaperlayerView exact revision on arXiv

Source-linked AI summary

DeiT III: Revenge of the ViT

Hugo Touvron, Matthieu Cord, Hervé Jégou

arXiv:2204.07118v1cs.CV

TL;DR

Fully supervised ViT training lacked strong evidence on midsize datasets, complicating comparisons with BERT-like self-supervised methods. The paper introduces simplified supervised recipes centered on three-transform augmentation and related training choices. Across classification, transfer learning, and segmentation, the recipes substantially improve supervised ViT results and provide competitive baselines, including for large models trained with fewer resources.

  • Problem

    The paper addresses limited evidence on efficiently training vanilla ViTs supervised on midsize datasets such as ImageNet1k.

  • Method

    It develops supervised ViT recipes based on a ResNet-50 training approach, adding stochastic depth, LayerScale, lower-resolution pre-training, and three-transform augmentation.

  • Results

    The recipes improve supervised ViT performance across image classification, transfer learning, and semantic segmentation, reaching 85.2% top-1 accuracy for ViT-H on ImageNet1k.

  • Takeaways & Limitations

    The resulting supervised models provide stronger baselines for comparing new architectures and self-supervised training approaches.

  • Takeaways & Limitations

    ImageNet-V2 measurements are less robust because the test set has 10,000 rather than 50,000 images, increasing the standard deviation to 0.2%.

Abstract

from arXiv · show

A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT.

1 Introduction

The paper revisits supervised ViT training to test stronger recipes on ImageNet and downstream tasks, using simplified augmentation and optimization choices. The resulting models scale to larger capacities, reduce resource demands, and become competitive with self-supervised and newer architectural approaches.

  • Motivation: The paper addresses limited evidence on efficiently training vanilla ViTs supervised on midsize datasets such as ImageNet1k.It frames stronger supervised baselines as necessary for interpreting self-supervised gains and comparing optimization with architectural changes.
  • Training recipe: 3-Augment uses three simple transformations and performs better with ViT than usual automatic or learned augmentation such as RandAugment.The policy is inspired by augmentation used in self-supervised learning.
  • Training recipe: Simple Random Cropping outperforms Random Resize Cropping during pre-training on ImageNet-21k.
  • Results: 85.2% top-1 accuracy is achieved by ViT-H trained only on ImageNet1k, improving +5.1% over the best reported supervised ViT-H at 224×224.
  • Results: 84.9% at 224×224 is attained by a billion-parameter, 52-layer ViT-H without hyper-parameter adaptation, +0.2% above the corresponding same-setting ViT-H.
  • Efficiency: The procedure halves GPU requirements and training time for ViT-H without sacrificing performance by pre-training at lower resolution.Lower resolution also reduces peak memory.
  • Results: For ViT-B and ViT-L, supervised training is on par with default-setting BERT-like self-supervised approaches for classification and semantic segmentation.The comparison uses the same annotation level and fewer epochs.
  • Results: The trained vanilla ViT closes the gap with recent architectures, offers better compute/performance trade-offs, and generalizes better to ImageNet-V2 than most prior works.

2 Related work

Prior work established ViTs as flexible alternatives to convolutional networks but highlighted their data and training demands. Related research focused on training recipes, BERT-like pre-training, and increasingly simplified augmentation strategies.

  • Vision Transformers: ViTs are used across classification, detection, segmentation, and video analysis, but their flexibility typically requires larger datasets or adapted training when data is scarce.
  • Training procedures: The original ViT training procedure was ineffective against convnets from scratch on ImageNet, while DeiT made performance comparable by adapting training.
  • BERT pre-training: BERT-like self-supervised approaches such as BeiT and MAE outperformed fully supervised ViT training, especially for larger models, before this paper’s revised supervised recipe.
  • BERT pre-training: BeiT’s best performance required full fine-tuning on ImageNet-21k followed by ImageNet-1k, suggesting that substantial labeled data supports its potential.
  • Data augmentation: The paper extends simplified augmentation beyond procedures optimized for convnets by using only three transformations sampled uniformly.

3 Revisit training & pre-training for Vision Transformers

The paper revises supervised ViT training with regularization, loss, and augmentation changes, including 3-Augment and crop strategies tailored to dataset scale. It also reports simpler cropping as advantageous for ImageNet-21k, while stronger RRC regularization helps ImageNet-1k.

  • 3.2 Data-augmentation: 3-Augment applies one of grayscale, solarization, or Gaussian blur uniformly, alongside color jitter and horizontal flipping.The approach is inspired by self-supervised learning and limits the main augmentation choices to three transformations.
  • 3.3 Cropping: On ImageNet-1k, removing Random Resized Crop lowers ViT-S top-1 accuracy by −0.9%.The stronger diversity and regularization of RRC are beneficial for this more overfitting-prone setting.
  • 3.3 Cropping: Simple Random Crop preserves more image content and aspect ratio, whereas Random Resized Crop provides greater crop diversity and regularization.RRC can create scale and aspect-ratio discrepancies and may exclude the labeled object; SRC resizes the shortest side, pads four pixels, and samples a square crop.
  • 3.3 Cropping: Simple Random Crop is more effective than Random Resized Crop during ImageNet-21k pre-training, where overfitting is less likely.SRC reduces train-test discrepancies and makes label-preserving crops more likely, translating into a significant performance gain.

4 Experiments

The experiments evaluate the revised ViT training procedure across image classification, ImageNet-21k pre-training, transfer learning, and segmentation-related settings. The default protocols specify training durations, batch size, resolutions, and fine-tuning choices.

  • Evaluation scope: The evaluation covers ImageNet-1k classification, ImageNet-V2, ImageNet-21k pre-training, six transfer-learning benchmarks, and downstream segmentation or fine-grained classification.ImageNet-1k validation is used alongside ImageNet-V2 to assess overfitting and generalization.
  • Default settings: ImageNet-1k-only training uses 400 epochs, batch size 2048, and 224 × 224 resolution by default.The paper recommends lower-resolution training when targeting 224 × 224 inference, despite the default evaluation setting.
  • Default settings: ImageNet-21k pre-training uses 90 epochs at 224 × 224, followed by 50 ImageNet-1k fine-tuning epochs at either 224 × 224 or 384 × 384.Two fine-tuning resolutions are considered after pre-training.

4.2 Ablations

The ablations identify simpler augmentation, lower-resolution training, and longer schedules as important components of the proposed supervised ViT recipe. These choices improve accuracy while reducing overfitting and training cost.

  • Training duration: Training beyond the 400-epoch baseline continues to improve ViT performance rather than saturating as rapidly as the DeiT procedure.Figure 5 varies the number of epochs for ImageNet-1k-only training at 224×224.
  • Data augmentation: 3-Augment is more effective for ViT than existing learned augmentation methods while using a simpler policy.For ResNet-50, existing policies are similar or better, so the advantage is architecture-dependent.
  • Data augmentation: Simple Random Cropping is more effective than Random Resize Cropping for ImageNet-21k pre-training.The ImageNet-21k ablation also varies LayerScale, Mixup, 3-Augment, and pre-training duration.
  • Training resolution: Lower-resolution training reduces the cost of processing large ViTs and makes billion-parameter models easier to train.The reduction in patches lowers training demands while maintaining or increasing performance.
  • Comparison with prior recipes: +3.0% is the reported ViT-L improvement with ImageNet-21k pre-training over the best prior approach.For ImageNet-1k-only training, the reported improvements are +2.1% for ViT-H and +4.3% over the best approach without EMA.

4.3 Image Classification

The proposed recipes outperform prior supervised ViT training procedures across ImageNet-1k and ImageNet-21k settings, with favorable generalization and comparable performance to selected BERT-like pre-training approaches.

  • Generalization: The models overfit less than the other considered models when comparing ImageNet validation accuracy with ImageNet-v2 accuracy.The paper uses the validation-to-v2 comparison to assess generalization without fine-tuning.
  • ImageNet-21k: The ImageNet-21k findings are similar to those obtained with ImageNet-1k-only training.Table 8 compares ViTs pre-trained on ImageNet-21k and then fine-tuned on ImageNet-1k.
  • Comparison with self-supervision: For equivalent epochs, the supervised approach is comparable on ImageNet-1k and better on ImageNet-v2 and ADE20k segmentation than the compared BERT-like approaches.The comparison includes MAE and BeiT under the reported evaluation settings.
  • Caveat: The ImageNet-v2 measurements are less robust because they use 10,000 test images instead of 50,000 for ImageNet validation.This produces a higher reported standard deviation of 0.2%.
  • Scope of comparison: The comparison does not include PeCo or iBoT, which use auxiliary losses, multi-crop, or exponential moving averages.The omitted methods therefore represent untested alternatives within the paper’s comparison.

4.4 Downstream tasks and other architectures

The trained ViTs are evaluated on transfer learning, semantic segmentation, and alternative transformer architectures. Results support broad downstream utility, but the recipe’s effect depends on the architecture.

  • Transfer learning: Transfer learning is evaluated on six datasets using ViT models pre-trained on ImageNet-1k for 400 epochs at 224×224.Figure 6 additionally examines the effect of test-time crop ratio.
  • Transfer learning: Test-time crop ratio has a significant impact on iNaturalist transfer performance.The experiments vary crop ratio while evaluating transfer learning performance across the six datasets.
  • Semantic segmentation: Semantic segmentation is evaluated on ADE20k with UperNet using 160k training iterations and single-scale and multi-scale testing.The ADE20k dataset contains 20k training and 5k validation images across 150 categories.
  • Other architectures: The recipe improves some transformer architectures, is neutral for TNT, and decreases performance for Swin.The paper reports these effects on ImageNet validation, ImageNet-real, and ImageNet-v2 measurements.

5 Conclusion

The paper presents improved supervised-training baselines for ViTs and shows that large models can be trained with fewer resources without hurting accuracy. These baselines are intended for comparisons with new architectures and self-supervised methods.

  • Contribution: The contribution is an improved supervised baseline for comparing new vision architectures and self-supervised training approaches.The stated goal is to support progress on learning foundation models for multiple tasks.
  • Resource efficiency: A one-billion-parameter ViT can be trained with four nodes of eight GPUs without hurting accuracy.The paper connects this result to insights for training larger ViTs with reduced resources.

A Experimental details

The experiments specify higher-resolution fine-tuning, model-size-dependent stochastic depth, and evaluation on six transfer-learning datasets.

  • Fine-tuning at higher resolution: Fine-tuning uses higher resolution to address the train-test resolution discrepancy after ImageNet-1k pre-training at 224 × 224.The procedure uses 20 epochs, learning rate 1 × 10^-5, batch size 512, and weight decay 0.1.
  • Stochastic depth: Stochastic depth drop rates are adapted according to model size.Table 13 reports values for 400-epoch ImageNet-1k and 90-epoch ImageNet-21k training.
  • Transfer learning: Transfer-learning experiments evaluate models pre-trained at 224 × 224 on ImageNet-1k across 6 datasets.The datasets are listed in Table 14.

B Additional Ablations

Additional ablations examine training duration and longer-training adjustments, finding no observed saturation as the number of ImageNet-1k epochs increases.

  • Number of training epochs: The ImageNet-1k ablation does not observe saturation when increasing the number of training epochs.Table 15 reports the performance impact of training duration.
  • Number of training epochs: For longer training, weight decay increases from 0.02 to 0.05.This adjustment accompanies the longer-training schedule.
  • Number of training epochs: For longer training, stochastic depth drop-rate increases by 0.05 every 200 epochs to prevent overfitting.The schedule also increases weight decay from 0.02 to 0.05.
Loading 2204.07118v1…