PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting ResNets: Improved Training and Scaling Strategies

Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph

arXiv:2103.07579v1cs.CV

TL;DR

The paper addresses the difficulty of separating architectural effects from training and scaling effects in vision models. It revisits ResNets with systematic training and scaling studies, achieving strong accuracy and efficiency across image, transfer, and video tasks. The results support using revised ResNets as strong baselines while showing that scaling choices depend on the training regime.

  • Problem

    Vision-model comparisons often conflate architectural changes with simultaneous changes in training methodology and hyperparameters.

  • Method

    The paper empirically studies ResNets by combining modern training methods, minor architectural changes, and scaling strategies across model sizes and regimes.

  • Results

    ResNet-RS achieves competitive accuracy with 1.7x - 2.7x speedups over EfficientNets on TPUs, while results also extend to transfer learning and video classification.

  • Takeaways & Limitations

    Revised ResNets provide strong baselines, and scaling choices should account for training regime and overfitting.

  • Takeaways & Limitations

    The authors are unsure how their scaling strategies apply to tasks requiring larger image resolutions, such as detection and segmentation.

Abstract

from arXiv · show

Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies. Our work revisits the canonical ResNet (He et al., 2015) and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended (Tan & Le, 2019). Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than EfficientNet NoisyStudent. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.

1. Introduction

The paper argues that vision-model performance reflects architecture, training, and scaling together, so these factors should be disentangled. Revisiting ResNets with improved training and scaling yields strong accuracy, speed, transfer, and video results.

  • Training and architecture: Training methods alone improve a canonical ResNet from 79.0% to 82.2% top-1 ImageNet accuracy.Two minor architectural changes further increase accuracy to 83.4%.
  • Training and architecture: +3% top-1 accuracy is achieved through regularization techniques without changing the model architecture.The study examines regularization methods and their interactions.
  • Scaling strategies: ResNet-RS scales depth when overfitting can occur and increases image resolution more slowly than prior recommendations.Width scaling can be preferable in other training regimes.
  • ResNet-RS: 1.7x - 2.7x faster on TPUs and 2.1x - 3.3x faster on GPUs, ResNet-RS outperforms EfficientNets on the speed-accuracy Pareto curve.The models use improved training and scaling strategies with only minor architectural changes.
  • ResNet-RS: 86.2% top-1 ImageNet accuracy is achieved by semi-supervised ResNet-RS using an additional 130M pseudo-labeled images.It is 4.7x faster on TPUs and 5.5x faster on GPUs than the corresponding EfficientNet-NoisyStudent.
  • Generalization: ResNet checkpoints rival or outperform self-supervised representations on downstream tasks, while 3D ResNet-RS improves top-1 Kinetics-400 accuracy by 4.8%.The improved training strategies are tested across transfer learning and video classification.

2. Characterizing Improvements on ImageNet

ImageNet progress comes from four broad axes: architecture, training and regularization, scaling, and additional training data. These axes encompass both model-design changes and changes to optimization, capacity, resolution, and data.

  • Architecture: Architecture improvements include VGG, ResNet, Inception, ResNeXt, automated search models, and visual adaptations of self-attention.The literature has advanced through both standard convolutional architectures and alternatives based on attention.
  • Training and regularization: Training and regularization methods improve generalization when models are trained for more epochs.Examples include dropout, label smoothing, stochastic depth, DropBlock, data augmentation, and improved learning-rate schedules.
  • Scaling strategies: Scaling model width, depth, and image resolution is another route to improving vision-model quality.EfficientNet formalized this with a compound scaling rule balancing depth, width, and resolution.
  • Additional training data: Additional labeled, weakly labeled, or unlabeled data can further improve accuracy through pretraining or semi-supervised learning.Noisy Student uses pseudo-labels on an extra 130M unlabeled images.

3. Related Work on Improving ResNets

Prior ResNet improvements commonly combine architectural modifications with modern training methods, complicating comparisons. This work instead emphasizes lightweight architecture changes while focusing on training and scaling strategies.

  • Prior ResNet improvements: Improved training methods combined with architectural changes have repeatedly produced competitive ImageNet performance for ResNets.Examples include ResNet modifications using label smoothing, mixup, and other training improvements.
  • This work: 1.7x - 2.7x faster on TPUs, the resulting ResNets improve the speed-accuracy Pareto curve relative to EfficientNets.The scaling improvements are described as orthogonal to earlier training and architectural methods and expected to be additive.

4. Methodology

The methodology combines a ResNet backbone with two established architectural modifications and a modern training recipe. It evaluates regularization, augmentation, optimization, and hyperparameters under a controlled setup.

  • Architecture: The study uses ResNet-D and Squeeze-and-Excitation as its two widely used architectural changes.These modifications are applied while retaining the ResNet-centered experimental framework.
  • Architecture: ResNet-D modifies the stem and downsampling blocks through smaller convolutions, stride changes, average pooling, and removal of a max-pooling layer.The passage enumerates four adjustments to the original ResNet architecture.
  • Architecture: Squeeze-and-Excitation reweights channels using cross-channel interactions from average-pooled signals across the feature map.Experiments use a Squeeze-and-Excitation ratio of 0.25.
  • Training methods: The training methodology studies regularization and augmentation techniques used in modern classification and semi/self-supervised learning.Regularization includes weight decay, label smoothing, dropout, and stochastic depth.
  • Training methods: The setup trains for 350 epochs and differs from EfficientNet through cosine decay, RandAugment, and Momentum optimization.These choices are presented as small differences from the EfficientNet setup.
  • Training methods: RandAugment applies random image transformations independently to each training image as an additional regularizer.The transformations include translation, shearing, and color distortions.
  • Hyperparameter tuning: Hyperparameters are selected using a held-out minival-set containing 2% of the ImageNet training set.This corresponds to 20 shards out of 1024.

5. Improved Training Methods

Improved training methods account for most of the measured ResNet improvement, while combining regularization techniques requires reducing weight decay to avoid over-regularization.

  • 5.1. Additive Study of Improvements: Training methods alone improve baseline ResNet-200 accuracy from 79.0% to 82.2% (+3.2%) without architectural changes.
  • 5.1. Additive Study of Improvements: Adding Squeeze-and-Excitation and ResNet-D further increases accuracy to 83.4%.
  • 5.1. Additive Study of Improvements: Training methods provide 3/4 of the total improvement in the additive study.
  • 5.2. Importance of decreasing weight decay when combining regularization methods: EfficientNet-B6 performance may be 0.1-0.3% top-1 higher if evaluated with RandAugment, making the comparison more nuanced.
  • 5.2. Importance of decreasing weight decay when combining regularization methods: Combining dropout and stochastic depth with other regularization can reduce performance unless weight decay is decreased.

6. Improved Scaling Strategies

Scaling outcomes depend on training regime, model dimensions, and image resolution. The study therefore favors depth scaling when overfitting can occur and slower image-resolution growth, while cautioning against small-regime extrapolation.

  • 6. Improved Scaling Strategies: The scaling search trains widths [0.25,0.5,1.0,1.5,2.0], depths [26,50,101,200,300,350,400], and resolutions [128,160,224,320,448] for 350 epochs.
  • 6. Improved Scaling Strategies: The best-performing scaling strategy depends strongly on the training regime, including epoch count, model size, and image resolution.
  • 6.1. Strategy #1 - Depth Scaling in Regimes Where Overfitting Can Occur: Depth scaling significantly outperforms width scaling across image resolutions after 350 epochs, while width scaling is better after 10 epochs.
  • 6.1. Strategy #1 - Depth Scaling in Regimes Where Overfitting Can Occur: Width scaling can overfit and sometimes hurt performance even with increased regularization, whereas depth scaling introduces fewer parameters.
  • 6.2. Strategy #2 - Slow Image Resolution Scaling: Image-resolution scaling shows diminishing returns at larger resolutions, motivating slower resolution increases than prior strategies.
  • 6. Improved Scaling Strategies: Scaling rules derived from small models or few epochs can fail to generalize to larger models or longer training iterations.
  • 6. Improved Scaling Strategies: For new tasks, the authors recommend evaluating a small subset of models across scales for the full training duration.
  • 6. Improved Scaling Strategies: ResNet-RS and EfficientNet-RS use these strategies and achieve significant speed-ups over EfficientNets.

7. Experiments with Improved Training and Scaling Strategies

Experiments show that improved training and scaling make ResNet-RS competitive with EfficientNet across image, semi-supervised, transfer-learning, and video settings, often with better efficiency.

  • ResNet-RS efficiency: 1.7x - 2.7x faster on TPUs, ResNet-RS matches EfficientNets’ ImageNet performance on the speed-accuracy Pareto curve.ResNet-RS also uses less memory despite having more parameters and FLOPs.
  • EfficientNet re-scaling: Re-scaled EfficientNet-RS improves the original EfficientNet’s speed-accuracy Pareto efficiency, while ResNet-RS still outperforms EfficientNet-RS.The improvement uses slower image-resolution scaling without changing width or depth.
  • Semi-supervised learning: 4.7x faster on TPUs than EfficientNet-B5 Noisy Student, ResNet-RS achieves similar ImageNet accuracy using the same 130M pseudo-labeled images.The corresponding GPU speedup is 5.5x.
  • Transfer learning: Improved supervised representations outperform SimCLR on 5/10 downstream tasks and SimCLRv2 on 8/10 tasks.They also substantially outperform standard supervised ResNet representations.
  • Video classification: Training strategies improve Kinetics-400 accuracy from 73.4% to 77.4% (+4.0%), with architectural changes raising it further to 78.2% (+0.8%).Most of the improvement is obtained without architectural changes.

8. Discussion

The discussion argues that training and scaling should be controlled separately from architecture because their effects depend on tasks, datasets, training regimes, and hardware.

  • Comparing architectures: Training methods can be more task-specific than architectures, making bundled architectural and training improvements difficult to compare fairly.For example, data augmentation is more helpful on small datasets.
  • Hardware-aware evaluation: FLOPs and parameter counts are not sufficient proxies for practical efficiency because latency and memory depend on operations, activations, compiler optimizations, and hardware.The authors recommend measuring latency and memory on the target software-hardware stack.
  • Scaling across tasks: The best scaling strategy depends on training regime and overfitting: depth works well for 350 epochs on ImageNet, while width is preferable for short regimes such as 10 epochs.The authors are unsure how these strategies apply to tasks requiring larger image resolutions.
  • Practical guidance: Practitioners should start with a hardware-efficient simple architecture, train models to convergence across resolutions, widths, and depths, and construct a Pareto curve.This differs from allocating substantial compute to identify an initial architecture before scaling it.

9. Conclusion

The conclusion presents ResNet-RS as evidence that a simple, modernized ResNet remains a strong vision baseline when training and scaling are updated and comparisons are methodologically consistent.

  • Conclusion: Updating the de facto vision baseline with modern training and scaling reveals the durability of the ResNet architecture.The authors emphasize that simple architectures can set strong baselines for state-of-the-art methods.
  • Conclusion: ResNet-RS models achieve 1.7x - 2.7x speedups across the EfficientNet Pareto curve on TPUs.The models in this curve are trained for 350 epochs with the ResNet-RS improvements.
  • Conclusion: The ResNet-RS family combines improved scaling, improved training, ResNet-D modifications, and Squeeze-and-Excitation.Architectural details vary with model depth, especially in the c3 and c4 stages.

C. ResNet-RS Training and Regularization Methods

ResNet-RS uses a simplified training recipe together with depth-dependent block layouts and established architectural modifications to regularize and scale ResNets.

  • Training recipe: ResNet-RS closely matches EfficientNet’s training setup while simplifying cosine learning-rate decay and Momentum choices.These choices reduce the number of associated hyperparameters compared with exponential decay and RMSProp.
  • Training recipe: All ResNet-RS models train for 350 epochs with weight decay 4e-5, EMA 0.9999, RandAugment, label smoothing 0.1, and cosine learning-rate decay.The learning rate warms up to 0.1/B before decaying to zero.
  • Architecture: ResNet-RS combines ResNet-D and Squeeze-and-Excitation with the revised training and scaling strategies.Figure 6 presents the architecture under a 224×224 input resolution.
  • Architecture scaling: Depth scaling mainly adds blocks in c3 and c4, limiting overfitting associated with adding blocks to c5.ResNet depths 50 through 200 retain standard block configurations.

E. Scaling Analysis Regularization and Model Details

The section specifies regularization, scaling, block allocation, fine-tuning, and downstream-task protocols used to evaluate ResNet variants. These details span training durations, augmentation choices, model configurations, and task-specific evaluation setups.

  • Regularization: Filter scaling uses multiplier-dependent dropout values across image resolutions and depths.Filter scaling is defined relative to the number of filters in the original ResNet architecture.
  • Regularization: Regularization differs by training duration: 10- and 100-epoch models omit RandAugment, Dropout, Stochastic Depth, and Label Smoothing, while using weight decay 4e-5.The 350-epoch regime uses additional regularization settings, including RandAugment and stochastic depth.
  • Model Details: ResNet-300 and ResNet-400 use block allocations [4-36-54-4] and [6-48-72-6], respectively.ResNet-101 and ResNet-200 use allocations specified separately in Table 11.
  • Fine-tuning: Fine-tuning initializes the backbone from a pretrained model, randomly initializes remaining layers, and searches learning rates and training steps end-to-end.Weight decay is set to 1e-4 because different values did not make a big difference.
  • Downstream Tasks: CIFAR-100 evaluation resizes images to 256×256, replaces the head with a 101-class linear layer, and searches training steps from 5000 to 20000.The protocol reports top-1 accuracy and searches learning rates from 0.005 to 0.32.
  • Downstream Tasks: PASCAL segmentation reports mIoU using 512 × 512 crops with scale jittering, while PASCAL detection reports AP50 using Faster R-CNN and FPN layers.The segmentation and detection protocols use different task heads and image resolutions.

G. Video Classification Experimental Details

The video experiments apply established spatial-temporal sampling and inference protocols, then evaluate a baseline against an improved training configuration under standardized latency profiling.

  • Training Protocol: Video inputs use random 224×224 crops or horizontal flips and 32-frame clips with temporal stride 2.These protocols follow prior video-classification training and inference procedures.
  • Improved Training: The improved training configuration combines dropout, label smoothing, stochastic depth, EMA, smaller weight decay, a 350-epoch schedule, and scale jittering.Scale jittering replaces RandAugment, and stochastic depth is reduced to 0.1 when scale jittering is applied.
  • Profiling Setup: Training latencies are measured on 8 TPUv3 cores with bfloat16 or on one Tesla-V100 with float32, using normalized batch sizes.Batch sizes are divided when necessary to fit accelerator memory, and reported TPU latency is normalized to batch size 1024.
Loading 2103.07579v1…