Source-linked AI summary
Better plain ViT baselines for ImageNet-1k
Lucas Beyer, Xiaohua Zhai, Alexander Kolesnikov
TL;DR
ImageNet-1k remains an important testbed, yet simple effective ViT baselines are limited. This paper revisits minor training details while retaining the original architecture, showing that the resulting setup matches key ResNet and ViT baselines with modest compute.
Problem
ImageNet-1k remains a key computer-vision testbed, motivating a simple and effective ViT baseline beyond the focus on large-scale pre-training.
Method
The paper retains the original ViT architecture and revisits a few minor training details, including batch size, pooling, position embeddings, RandAugment, and Mixup.
Results
The setup matches the seminal ResNet50 and more modern ResNet and ViT training baselines; 80% accuracy is reached in less than a day with 300 epochs.
Takeaways & Limitations
A collection of almost trivial changes can accumulate to an important overall improvement while preserving a simple ViT baseline.
Takeaways & Limitations
The baseline omits architectural changes, additional regularizers, advanced optimization, extra augmentations, high-resolution fine-tuning, checkpoint averaging, and knowledge distillation.
Abstract
from arXiv · showhide
It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.
1. Introduction
ImageNet-1k remains an important testbed, yet the original ViT work emphasized large-scale pre-training. This note introduces a simple baseline that preserves ViT simplicity while achieving competitive results.
- ImageNet-1k remains a key computer-vision testbed, motivating a simple and effective baseline.
- The baseline stays true to the original ViT’s simplicity while reaching competitive results against related and concurrent approaches.
2. Experimental setup
The experiments use the original ViT architecture on ImageNet-1k, focusing on ViT-S/16 and standard augmentations. The setup also reserves 1% of training data for minival evaluation.
- The study uses ImageNet-1k for both training and evaluation while retaining the original ViT architecture.
- The experiments focus on ViT-S/16 as a tradeoff between iteration speed on common hardware and final accuracy.
- The setup applies inception crop, random horizontal flips, RandAugment, and Mixup at 224px² resolution.
- The training split uses 99% of the data, reserving 1% for minival to discourage validation-set design selection.
3. Results
A plain ViT with small training changes matches important ResNet and ViT baselines. The 90-epoch run takes 6h30 on TPUv3-8, while 300 epochs reach 80% accuracy in less than a day.
- A simple standard ViT matches the seminal ResNet50 baseline at 90 epochs and more modern ResNet and ViT training setups.
- 80% accuracy is reached in less than a day with 300 epochs of training.
- The revised setting changes batch size, pooling, positional embeddings, RandAugment, and Mixup relative to earlier ViT setups.
- The baseline omits architectural changes, additional regularizers, advanced optimization, extra augmentations, high-resolution fine-tuning, checkpoint averaging, and knowledge distillation.
- The ablation shows that individually minor modifications accumulate into an important overall improvement, while the classification-head choice makes no significant accuracy difference.
4. Conclusion
The conclusion emphasizes continued pursuit of simplicity. The supplied conclusion passages contain no additional substantive result beyond that principle.
- The paper concludes that striving for simplicity remains worthwhile.
A. big vision experiment configuration
The recommended configuration retains image and label fields, applies JPEG decoding, cropping, horizontal flipping, and RandAugment, and evaluates multiple classification splits.
- The configuration keeps the image and label fields for processing.
- Training preprocessing decodes JPEG images, performs inception-style cropping to 224 pixels, applies horizontal flipping, and uses RandAugment with parameters (2,10).
- Evaluation preprocessing applies a central crop at 224 pixels.
- The configuration evaluates the minival, val, real, and v2 classification splits.
- Listing 1 presents the full recommended configuration.