PaperlayerView exact revision on arXiv

Source-linked AI summary

ResNet strikes back: An improved training procedure in timm

Ross Wightman, Hugo Touvron, Hervé Jégou

arXiv:2110.00476v1cs.CVcs.LG

TL;DR

ResNet-50 performance is difficult to assess when architecture and training recipes evolve independently. This paper develops optimized training procedures and establishes a new state of the art for vanilla ResNet-50 while offering lower-resource alternatives.

  • Problem

    Evidence was limited on how far vanilla ResNet-50 performance could be improved through training alone, complicating fair architecture comparisons.

  • Method

    The paper develops three resource-performance training procedures for vanilla ResNet-50, integrating recent ingredients, hyperparameter search, and a Mixup/CutMix-compatible binary cross-entropy objective.

  • Results

    Procedure A1 surpasses the current state of the art for vanilla ResNet-50 on ImageNet at 224×224, while A2 and A3 achieve lower but still high performance with fewer resources.

  • Takeaways & Limitations

    The procedures provide strong ResNet-50 baselines at different compute levels, supporting more informed comparisons between architectures and training procedures.

  • Takeaways & Limitations

    The authors do not claim the procedures are universal and state that architecture and training should be optimized jointly.

Abstract

from arXiv · show

The influential Residual Networks designed by He et al. remain the gold-standard architecture in numerous scientific publications. They typically serve as the default architecture in studies, or as baselines when new architectures are proposed. Yet there has been significant progress on best practices for training neural networks since the inception of the ResNet architecture in 2015. Novel optimization & data-augmentation have increased the effectiveness of the training recipes. In this paper, we re-evaluate the performance of the vanilla ResNet-50 when trained with a procedure that integrates such advances. We share competitive training settings and pre-trained models in the timm open-source library, with the hope that they will serve as better baselines for future work. For instance, with our more demanding training setting, a vanilla ResNet-50 reaches 80.4% top-1 accuracy at resolution 224x224 on ImageNet-val without extra data or distillation. We also report the performance achieved with popular models with our training procedure.

1 Introduction

The paper addresses the underdeveloped training of vanilla ResNet-50 by proposing stronger baselines that incorporate recent training advances and careful hyper-parameter adjustment. It evaluates these procedures across runs and architectures while examining stability, overfitting, and fair architecture comparisons.

  • Comparison methodology: Fair architecture comparisons require accounting jointly for architecture design and training setting because older reported results may use weaker recipes.The introduction recommends incorporating newer training ingredients into baselines and adjusting hyper-parameters to disentangle training improvements from architectural improvements.
  • Motivation: Reported ImageNet-1k-val performance for ResNet-50 ranges from 75.2% to 79.5%, motivating a focused effort to improve the baseline.The authors state that no prior effort specifically targeted extensive ingredient selection and hyper-parameter search for ResNet-50.
  • Contributions: The paper proposes three strong-baseline training procedures for vanilla ResNet-50 at 224 × 224 inference resolution, using 100, 300, or 600 epochs.Each variant adjusts its hyper-parameters and training ingredients.
  • Training procedure: Mixup and CutMix training minimizes binary cross entropy for each selected concept rather than using the usual cross-entropy loss.The procedure treats the task as multi-classification and assumes all mixed concepts are present in the synthesized image.
  • Evaluation: The study measures accuracy stability across runs with different seeds and compares ImageNet-val with ImageNet-V2 to examine overfitting.It also re-evaluates popular architectures and argues that a shared training procedure alone does not establish fair architectural comparisons.

2 Related work

Image classification is a central computer-vision benchmark whose ImageNet-pretrained models support downstream tasks. ResNet remains a widely used reference and baseline, while timm and modern training practices provide broader model and recipe implementations.

  • Image classification: Image classification benchmarks computer-vision progress, and ImageNet-pretrained models are used for downstream detection and segmentation.Progress in image classification generally translates to progress on these downstream tasks.
  • timm: timm provides implementations of popular image-classification models, training methods, and pre-trained weights.The library also includes data augmentations, regularization techniques, optimizers, and learning-rate schedules.
  • ResNet: ResNet remains one of the most popular image-classification architectures and serves as a reference architecture or baseline for new-model papers.It was a noteworthy improvement when introduced and continues to support architectural analysis.
  • ResNet: Modernized ResNet training procedures improve over the original model, but improving the ResNet-50 baseline was generally not their main objective.Such modernization enables more direct comparisons with models or methods using more elaborate training procedures.
  • Training recipes: Image-classification training recipes have evolved substantially, including longer progressive learning-rate schedules and jointly increased epochs and batch sizes with mixed precision.These changes better leverage powerful GPUs; AdamW and Lamb are also popular optimizer choices.

3 Training Procedures

The section presents three ResNet-50 training procedures targeting 224 × 224 performance, with different resource costs and schedules. Their recipes combine modern augmentation, multi-label BCE loss, schedule-dependent regularization, and LAMB optimization for large batches.

  • Training procedures: Procedure A1 targets the best ResNet-50 performance with 600 epochs and 4.6 days of training on one node with 4 V100 32GB GPUs.
  • Training procedures: Procedure A2 uses a 300-epoch schedule comparable to modern procedures such as DeiT, with batch size 2048 and recipe-specific choices.
  • Training procedures: Procedure A3 aims to outperform the original ResNet-50 procedure using 100 epochs, batch size 2048, and 15h on 4 V100 16GB GPUs.
  • Recipe ingredients: The procedures combine Random Resized Crop, horizontal flip, timm RandAugment, Mixup, and CutMix, treating mixed concepts as jointly present under binary cross-entropy.
  • Recipe ingredients: Regularization varies across procedures through weight decay, label smoothing, Repeated-Augmentation, and stochastic depth, while LAMB supports large batches such as 2048.

4 Experiments

The experiments show that the improved training procedures produce strong results across architectures and downstream tasks, while accuracy varies with random factors, evaluation datasets, and training choices. They also demonstrate that architectural comparisons can reverse depending on the training procedure and benchmark.

  • Training procedures: A1 surpasses the state of the art on ImageNet with vanilla ResNet-50 at 224×224, while A2 and A3 provide lower performance with fewer resources.The procedures are summarized as competitive alternatives with different resource requirements.
  • Cross-architecture performance: The procedures improve several architectures without hyper-parameter adaptation, although deeper and larger models require more regularization.Table 3 reports training time, GPU count, and peak memory alongside ImageNet classification performance.
  • Sensitivity and overfitting: The standard deviation is around 0.1 on ImageNet-val and 0.23 on ImageNet-V2, while the seed-0 weights overestimate mean performance by about +0.13%.ImageNet-V2 has greater variance because it contains 10000 rather than 50000 images.
  • Sensitivity and overfitting: The best checkpoint reaches close to 80.0% with A2, but ImageNet-val and ImageNet-V2 performance show limited correlation across random seeds.One outlying seed achieves 68.5% top-1 accuracy on ImageNet-V2 despite average ImageNet-val performance.
  • Transfer learning: A1 performs best overall on downstream tasks, whereas A3 is significantly inferior and A2 is similar to PyTorch default after fine-tuning on some datasets.A3’s weaker downstream performance may relate to its lower 160×160 training resolution.
  • Comparing architectures: ResNet-50 appears better than DeiT-S with A2 on ImageNet-val, whereas DeiT-S appears better with T2 and on ImageNet-V2, showing comparisons depend on procedure and benchmark.Both architectures were optimized substantially for ImageNet-val, complicating direct comparisons.

5 Ablations

The ablations identify learning rate, weight decay, BCE, repeated augmentation, stochastic depth, smoothing, and augmentation choices as important components of the training procedures. They also show that A1 and A2 benefit from larger test resolutions, while some ingredients interact with schedules and other hyper-parameters.

  • Main ingredients and hyper-parameters: The ablations focus on the intermediate A2 training procedure as a compromise between compute cost and accuracy.The optimizer is difficult to isolate because it strongly interacts with other choices and hyper-parameters.
  • Main ingredients and hyper-parameters: 5.10−3 leads to the best performance, but increasing learning rate further raises divergence risk; weight decay typically ranges from 0.02 to 0.03.Weight decay is sensitive and can interact with other regularization, with significant differences sometimes observed between 0.02 and 0.03.
  • Loss: Moving from BCE to vanilla CE significantly reduces performance because BCE supports Mixup/Cutmix as a multi-class 1-vs-all problem.Cross-entropy instead implicitly enforces probabilities that sum to 1.
  • Augmentation: Repeated augmentation provides a small boost, but can be neutral or detrimental with shortest schedules or higher Mixup values.It was best included in the most accurate A1 and A2 procedures.
  • Stochastic Depth & Smoothing: Stochastic depth improves A2 but not A3, while label smoothing is ineffective at 300 epochs and positive only for A1’s 600-epoch schedule.These conclusions hold with other hyper-parameters and ingredients fixed.
  • Augmentation: 79.72% – std 0.1 is the average top-1 accuracy over 100 seeds, and every tested augmentation modification falls below it.The modifications change Mixup or RandAugment parameters, and each affects the measured score.
  • Evaluation at other resolutions: A1 and A2 perform better at higher test resolutions, exhibiting the FixRes effect that does not appear for A3.The comparison uses ResNet-50 models trained with the three procedures.

6 Conclusion

The paper proposes new training procedures for vanilla ResNet-50, establishing a new state of the art while also providing two lower-compute alternatives. The authors emphasize that these procedures are not universal.

  • 6 Conclusion: The procedures establish a new state of the art for training vanilla ResNet-50 and include two alternatives requiring less compute power.The authors explored diverse procedures under different resource constraints but explicitly do not claim universality.

A Augmentations and Regularization in timm [50]

timm provides a broad set of image augmentations and regularization methods for ImageNet and other 2D image tasks, often with implementation changes beyond the originals. Its pipeline centers on Random Resized Crop and horizontal flipping, while adding modified RandAugment, Random Erasing, integrated Mixup/CutMix, and configurable regularization.

  • Data Augmentation: timm implements RandAugment, AutoAugment, AugMix, Random Erasing, and integrated Mixup/CutMix on a base of Random Resized Crop with horizontal flipping.These components are intended for ImageNet classification and other 2D image tasks.
  • RandAugment: RandAugment’s timm implementation adds an increasing mode that makes augmentation strength grow more intuitively with magnitude and enables the additional MSTD hyper-parameter.Solarize and posterize increase with M, while blending interpolation or extrapolation is randomly selected with strength increasing with M.
  • Implementation Details: timm reduces augmentation-induced shifts in image statistics by supporting specified normalization means, excluding Cutout by default, and offering normalized-image Random Erasing with per-pixel Gaussian noise.The Random Erasing variant uses Gaussian noise with mean 0 and standard deviation 1.0.
  • Regularization: Regularization options include weight decay, pre-classifier dropout, Stochastic-Depth through DropPath, and label smoothing across supported model architectures.Weight decay is available through native PyTorch or timm optimizers, and label smoothing uses a cross-entropy loss function.

B Alternative Training Procedures

The paper presents alternative training recipes that match or exceed existing ResNet-50 results and may suit different architectures, datasets, or tasks. These include RMSProp with EMA, SGD with Nesterov momentum and AGC, and AdamP-based training, with effectiveness depending on settings such as batch size and training length.

  • Alternative training procedures: Alternative recipes are introduced to match or exceed the best existing ResNet-50 results while supporting adaptation to different architectures, datasets, or tasks.The recipes are also used for pre-trained models in the timm library, including non-ResNet architectures.
  • Procedure B: Procedure B uses RMSProp with EMA weight averaging and step learning-rate decay.It adapts RandAugment and Random Erasing features from timm, with adjusted decay, weight decay, learning rate, and added per-pixel noise augmentation.
  • Procedure C: Procedure C uses SGD with Nesterov momentum, Adaptive Gradient Clipping, cosine learning-rate decay, and two variants based on repeated augmentation.Its recipe is based on NFNet training and combines heavy augmentation and regularization with AGC for stable large-batch training.
  • Procedure D: Procedure D uses AdamP with cosine learning-rate decay and binary cross-entropy, achieving 79.8 top-1 on ImageNet-1k in limited trials.The trials used a comparatively small batch size, and further experimentation is necessary.
  • Limitations: The RMSProp strategy varies with batch size, with larger batches of 1024-2048 often 0.1 to 0.3 top-1 below runs using batches of 256-768.It remains unclear whether further hyper-parameter adjustments and different learning-rate scaling can address this gap.
  • Procedure C: Procedure C trained for 600 epochs incurs an expected 0.15-0.2 top-1 drop for the same seed in both variants.The C1 and C2 variants differ most significantly in whether repeated augmentation is applied.
Loading 2110.00476v1…