Source-linked AI summary
Bag of Tricks for Image Classification with Convolutional Neural Networks
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, Mu Li
TL;DR
Image-classification research has underdocumented training refinements whose effects are often hidden in implementation details. The paper systematically evaluates these refinements and finds that combining them improves CNN accuracy and transfer-learning performance, while its large-batch scope is limited to single-machine training.
Problem
Training refinements affecting CNN accuracy are often only briefly documented as implementation details or source-code choices.
Method
The paper empirically evaluates minor architecture, preprocessing, loss-function, optimization, and learning-rate refinements across CNNs, datasets, and ablations.
Results
79.29% top-1 validation accuracy is achieved by ResNet-50 on ImageNet, up from 75.3%, while combined refinements improve ResNet-50, Inception-V3, and MobileNet.
Takeaways & Limitations
Improved pretrained models show advantages in transfer learning for object detection and semantic segmentation.
Takeaways & Limitations
The paper limits its large-batch methods to single-machine training, where batch sizes no more than 2K often provide good system efficiency.
Abstract
from arXiv · showhide
Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods. In the literature, however, most refinements are either briefly mentioned as implementation details or only visible in source code. In this paper, we will examine a collection of such refinements and empirically evaluate their impact on the final model accuracy through ablation study. We will show that, by combining these refinements together, we are able to improve various CNN models significantly. For example, we raise ResNet-50's top-1 validation accuracy from 75.3% to 79.29% on ImageNet. We will also demonstrate that improvement on image classification accuracy leads to better transfer learning performance in other application domains such as object detection and semantic segmentation.
1. Introduction
The paper argues that training and minor architecture refinements have been underdocumented despite materially affecting CNN accuracy. It evaluates these refinements across models and datasets, showing that combining them substantially improves results.
- Training procedure refinements received less attention than architecture advances and were often reported only as implementation details or source-code choices.
- The paper examines minor model and training refinements that improve accuracy while barely changing computational complexity.Examples include modifying convolution stride sizes and adjusting learning-rate schedules.
- The refinements are evaluated across multiple network architectures and datasets to measure their effect on final model accuracy.
- 79.29% top-1 validation accuracy is achieved by ResNet-50 on ImageNet, up from 75.3% after applying the tricks.The resulting ResNet-50 also outperforms newer architectures such as SE-ResNeXt-50.
2. Training Procedures
The paper establishes a minibatch-SGD baseline with explicit preprocessing, initialization, optimization, and validation procedures. These choices define the reference training pipeline for subsequent refinements.
- Minibatch SGD repeatedly samples b images, computes gradients, updates network parameters, and stops after K passes through the dataset.
- Validation resizes each image’s shorter edge to 256 pixels, takes a centered 224-by-224 crop, normalizes RGB channels, and applies no random augmentation.
- The baseline uses Xavier initialization for convolutional and fully connected weights, Nesterov accelerated gradient, 120 epochs, and learning-rate drops at epochs 30, 60, and 90.Training uses 8 Nvidia V100 GPUs with total batch size 256 and initial learning rate 0.1.
- The evaluation covers ResNet-50, Inception-V3, and MobileNet on the ISLVRC2012 dataset with 1.3 million training images and 1000 classes.Inception-V3 uses 299-by-299 input images.
3. Efficient Training
The paper studies techniques for faster training with lower precision and larger batches while preserving accuracy. It combines learning-rate, initialization, regularization, and mixed-precision heuristics, achieving faster training and a small accuracy gain.
- Larger batches reduce gradient noise but can degrade validation accuracy, motivating heuristics for scaling batch size on a single machine.
- Large-batch training: Linear learning-rate scaling increases the initial rate from 0.1 at batch size 256 to 0.1 × b/256 for batch size b.
- Large-batch training: Warmup starts training with a small learning rate before returning to the initial rate once training stabilizes.
- Large-batch training: Zero-γ initialization makes residual blocks initially return their inputs, while no-bias decay regularizes only convolutional and fully connected weights.
- Large-batch training: The single-machine scope limits the paper’s methods to batch sizes no more than 2K, whereas LARS is reported for batches beyond 16K.
- Efficient training results: 4.4 minutes per epoch is achieved for ResNet-50 with batch size 1024 and FP16, versus 13.3 minutes with batch size 256 and FP32.Stacking the large-batch heuristics slightly increases top-1 accuracy by 0.5% over baseline.
- Efficient training results: A 0.9% top-1 accuracy decrease from linear scaling alone is recovered by stacking the remaining three heuristics, while switching to FP16 at the end does not affect accuracy.
4. Model Tweaks
Model tweaks are minor architectural adjustments that can improve accuracy while barely changing computational complexity. The paper examines three ResNet modifications affecting downsampling blocks and the input stem, finding that their combination improves validation accuracy with limited overhead.
- Model Tweaks: Model tweaks change small architectural details, such as convolution strides, while often barely altering computational complexity.The paper uses ResNet to investigate how these adjustments affect model accuracy.
- ResNet-B: ResNet-B switches the strides of the first two path-A convolutions so the strided operation uses a 3 × 3 kernel instead of 1 × 1.This change is intended to avoid ignoring three-quarters of the input feature map while preserving path-A output shape.
- ResNet-C: ResNet-C replaces the input stem’s 7 × 7 convolution with three 3 × 3 convolutions.The replacement uses output channels of 32, 32, and 64, with stride 2 for the first two convolutions.
- ResNet-D: ResNet-D adds a 2×2 average-pooling layer before path B’s 1 × 1 convolution and changes that convolution’s stride to 1.The modification is designed to retain more input information while having little impact on computational cost.
- Experiment Results: Around 0.5% validation-accuracy improvement comes from ResNet-B, while replacing the 7 × 7 stem convolution contributes another 0.2%.Taking more information in path B adds another 0.3%, yielding a 1% improvement for ResNet-50-D over ResNet-50.
- Experiment Results: ResNet-50-D has the largest computational cost, but its FLOPs difference from ResNet-50 remains within 15% and its training throughput is only 3% slower.The four compared models have the same model size.
5. Training Refinements
The paper evaluates four training refinements—cosine decay, label smoothing, knowledge distillation, and mixup—and examines their effects individually and in combination. These refinements collectively improve accuracy across several architectures and datasets.
- Cosine Learning Rate Decay: Cosine decay decreases the learning rate gradually, remaining larger than step decay until the latter drops by 10x.The schedule decays slowly at the beginning, is nearly linear in the middle, and slows again near the end.
- Knowledge Distillation: Knowledge distillation adds a loss penalizing differences between teacher and student softmax outputs, using a temperature to smooth predictions.The teacher is a pretrained higher-accuracy model, such as ResNet-152 guiding ResNet-50.
- Mixup Training: Mixup forms training examples by weighted linear interpolation of two sampled examples and their labels, with λ drawn from Beta(α, α).The experiments use α = 0.2 and extend training from 120 to 200 epochs.
- Experiment Results: Stacking cosine decay, label smoothing, and mixup steadily improves ResNet, InceptionV3, and MobileNet, while distillation benefits ResNet but not the other two models.The authors attribute the negative effect on Inception-V3 and MobileNet to teacher–student family mismatch and differing prediction distributions.
- Experiment Results: Refinements improve top-5 accuracy consistently on both the validation and test sets of MIT Places365.
6. Transfer Learning
The paper tests whether improved image-classification models transfer to object detection and semantic segmentation. Higher classification accuracy corresponds to better detection performance, while segmentation responds differently across refinements.
- Evaluation Setup: The transfer-learning evaluation covers object detection with Faster-RCNN on PASCAL VOC and semantic segmentation with FCN on ADE20K.
- Object Detection: Higher ImageNet validation accuracy consistently corresponds to higher Faster-RCNN mAP on PASCAL VOC.
- Object Detection: 81.33% mAP on VOC is achieved by the base model with 79.29% ImageNet accuracy, outperforming the standard model by 4%.
- Semantic Segmentation: Cosine learning-rate scheduling improves FCN accuracy on ADE20K, whereas the other refinements produce suboptimal results.The reported metrics are pixel accuracy and mean intersection over union.
7. Conclusion
The paper finds that a dozen small training and architecture refinements consistently improve CNN accuracy, with further gains when combined. The resulting pretrained models also improve transfer learning for object detection and semantic segmentation.
- A dozen training and architecture tricks consistently improve accuracy across ResNet-50, Inception-V3, and MobileNet.The refinements modify architecture, preprocessing, loss functions, and learning-rate schedules.
- Stacking the tricks together leads to significantly higher accuracy than applying them individually.
- Improved pretrained models show strong transfer-learning advantages for both object detection and semantic segmentation.