PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientNetV2: Smaller Models and Faster Training

Mingxing Tan, Quoc V. Le

arXiv:2104.00298v3cs.CV

TL;DR

EfficientNetV2 addresses the need for faster, more parameter-efficient neural networks by combining training-aware architecture search and scaling with progressive learning and adaptive regularization. It achieves strong results across image datasets, training up to 11x faster and being up to 6.8x smaller than prior models.

  • Problem

    Training increasingly larger models and datasets makes efficient training important, while existing EfficientNets have slow large-image training, slow early depthwise convolutions, and sub-optimal uniform scaling.

  • Method

    The paper combines training-aware neural architecture search and scaling with Fused-MBConv operations, progressive image-size increases, and adaptive regularization.

  • Results

    EfficientNetV2 achieves strong results across ImageNet, CIFAR, Cars, and Flowers, training up to 11x faster and being up to 6.8x smaller than prior models.

  • Takeaways & Limitations

    EfficientNetV2 provides smaller, faster image-recognition models while progressive learning with adaptive regularization supports fast training and good accuracy.

  • Takeaways & Limitations

    The evaluation excludes models pretrained on non-public Instagram or JFT images and models using extra distillation or ensembles.

Abstract

from arXiv · show

This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.

1. Introduction

EfficientNetV2 combines training-aware neural architecture search and scaling to improve training speed and parameter efficiency. Its adaptive progressive learning further accelerates training while preserving or improving accuracy, yielding strong results across image-classification benchmarks.

  • EfficientNetV2 design: Training-aware NAS and scaling jointly optimize EfficientNetV2 for training speed and parameter efficiency.The search space adds operations such as Fused-MBConv, guided by identified EfficientNet bottlenecks.
  • EfficientNetV2 design: EfficientNet bottlenecks include slow training at very large image sizes, slow early-layer depthwise convolutions, and sub-optimal uniform stage scaling.These observations motivate an enriched search space and non-uniform architecture scaling.
  • Progressive learning: Adaptive progressive learning increases image size during training while adjusting regularization, speeding training without the accuracy drop caused by fixed regularization.The method jointly changes regularization such as dropout and data augmentation as image size changes.
  • Experimental results: 85.7% top-1 accuracy is achieved on ImageNet while training 3x - 9x faster and using up to 6.8x fewer parameters than previous models.The result uses the improved progressive learning method.
  • Experimental results: Up to 11x faster training speed and up to 6.8x better parameter efficiency are demonstrated on ImageNet, CIFAR, Cars, and Flowers than prior art.EfficientNetV2 also achieves strong results on CIFAR-10 and CIFAR-100.

2. Related work

Prior work has separately emphasized parameter efficiency, inference speed, progressive training, or NAS objectives such as FLOPs and inference efficiency. Unlike these approaches, this paper uses NAS to jointly optimize training and parameter efficiency.

  • Training and parameter efficiency: Parameter-efficiency research, including DenseNet and EfficientNet, targets better accuracy with fewer parameters, while RegNet and related models emphasize GPU or TPU inference speed.The passage contrasts parameter efficiency with recent work focused on training or inference speed.
  • Progressive training: Progressive training has been explored across GANs, transfer learning, adversarial learning, and language models, while progressive resizing specifically targets faster training but usually reduces accuracy.The related work spans dynamically changing training settings or networks across several application areas.
  • Neural architecture search: NAS has been applied to architecture design for classification, detection, segmentation, and hyperparameter optimization, but prior work mostly targets FLOPs or inference efficiency.The passage describes NAS as an automated network-design process and distinguishes its common optimization objectives.
  • Contribution: Unlike prior works, this paper uses NAS to optimize training and parameter efficiency.This directly states the paper’s distinction from earlier NAS and efficiency research.

3. EfficientNetV2 Architecture Design

EfficientNetV2 addresses EfficientNetV1’s training bottlenecks through training-aware neural architecture search and non-uniform scaling, combining MBConv and Fused-MBConv while constraining image size for faster training.

  • Training bottlenecks: Large training images increase memory usage, force smaller batches, and drastically slow EfficientNet training; smaller training images improve throughput.The paper uses FixRes by training at a smaller image size than inference, while reserving progressive image-size and regularization adjustment for later.
  • Training bottlenecks: Depthwise convolutions reduce parameters and FLOPs but often underutilize modern accelerators, especially in early layers.Fused-MBConv is introduced to better utilize mobile or server accelerators.
  • Scaling: EfficientNetV2 uses non-uniform scaling that gradually adds layers to later stages and restricts maximum image size to reduce memory consumption and slow training.This replaces EfficientNet’s uniform compound scaling, whose stages do not contribute equally to training speed and parameter efficiency.
  • NAS search: The training-aware NAS jointly optimizes accuracy, parameter efficiency, and training efficiency on modern accelerators.Its stage-based search space includes MBConv and Fused-MBConv, layer counts, and 3x3 or 5x5 kernel sizes.
  • EfficientNetV2 architecture: EfficientNetV2 extensively uses MBConv and newly added Fused-MBConv in early layers, while favoring smaller MBConv expansion ratios.These are identified as major distinctions from the EfficientNet backbone in the searched EfficientNetV2-S architecture.

4. Progressive Learning

Progressive learning improves efficiency by starting with small images and weak regularization, then increasing both image size and regularization strength as training proceeds. This adaptive schedule addresses the accuracy drop associated with changing image sizes and supports dropout, RandAugment, and Mixup.

  • Motivation: Changing image sizes during training often reduces accuracy, motivating a schedule that adjusts regularization strength alongside image size.The authors hypothesize that fixed regularization creates an imbalance across image sizes.
  • Method: The method starts training with smaller images and weak regularization, then gradually increases image size and strengthens regularization.The schedule makes early learning easier and faster before increasing difficulty through stronger dropout, RandAugment, and Mixup.
  • Method: Training is divided into M stages, with image size and regularization linearly interpolated from initial values to target values while inheriting weights between stages.The final stage uses target image size Se and regularization Φe.
  • Regularization: The progressive-learning procedure adjusts dropout rate γ, RandAugment magnitude ϵ, and Mixup ratio λ during training.These represent network-level regularization, per-image augmentation, and cross-image augmentation, respectively.

5. Main Results

EfficientNetV2 achieves faster training, strong accuracy, and improved parameter and inference efficiency than prior ConvNets and Transformers. It also transfers effectively beyond ImageNet and benefits substantially from ImageNet21k pretraining.

  • ImageNet results: EfficientNetV2-M matches EfficientNet-B7 accuracy while training 11x faster with the same computing resources.The models also outperform recent RegNet and ResNeSt models in accuracy and inference speed.
  • ImageNet results: 85.7% top-1 accuracy is achieved by EfficientNetV2-L, surpassing ViT-L/16(21k) despite using a convolutional architecture.The paper attributes this advantage to improved ConvNet design and training methods.
  • ImageNet21k results: 1.5% higher top-1 accuracy is achieved by EfficientNetV2-L(21k) than ViT-L/16(21k), with 2.5x fewer parameters, 3.6x fewer FLOPs, and 6x - 7x faster training and inference.The comparison is based on ImageNet21k pretraining followed by ImageNet ILSVRC2012 finetuning.
  • ImageNet21k results: Within two days, EfficientNetV2 pretraining on ImageNet21k completes using 32 TPU cores, despite the dataset containing 10x more data.The paper presents this as more efficient than training larger models on ImageNet.
  • Transfer learning: EfficientNetV2 models outperform previous ConvNets and Vision Transformers across CIFAR-10, CIFAR-100, Flowers, and Cars.On CIFAR-100, EfficientNetV2-L is 0.6% more accurate than prior GPipe/EfficientNets and 1.5% more accurate than prior ViT/DeiT models.

6. Ablation Studies

The ablations show that EfficientNetV2 remains faster and more parameter-efficient than EfficientNet under matched settings, while progressive learning and adaptive regularization improve training efficiency and accuracy across networks.

  • Performance with the same training: 17% fewer parameters and 37% fewer FLOPs are achieved by EfficientNetV2-M, which still outperforms EfficientNets under the same progressive-learning settings.Applying progressive learning to EfficientNet also reduces training time from 139h to 54h and improves accuracy from 84.7% to 85.0%.
  • Scaling Down: EfficientNetV2 models are generally faster than small-size EfficientNets while maintaining comparable parameter efficiency when scaled down without progressive learning.The comparison uses EfficientNet compound scaling and measures smaller models under identical non-progressive training conditions.
  • Progressive learning: 1.4x speedup is achieved for ResNet50(224), while progressive learning generally reduces training time and improves accuracy across the evaluated ResNet and EfficientNet networks.Larger default image sizes and more complex models obtain larger gains in accuracy and training efficiency.
  • Adaptive regularization: Adaptive regularization dynamically adjusts regularization according to image size and improves accuracy over vanilla progressive or random resizing with fixed regularization.The method is evaluated with progressively increasing image sizes and randomly sampled sizes, with TPU recompilation avoided by changing size every eight epochs.

7. Conclusion

EfficientNetV2 is a smaller, faster family of image-recognition networks optimized with training-aware NAS and model scaling. An improved progressive-learning method further accelerates training by jointly increasing image size and regularization.

  • EfficientNetV2 is a family of smaller and faster neural networks for image recognition.
  • Training-aware NAS and model scaling optimize EfficientNetV2 for stronger performance, faster training, and parameter efficiency.
  • Progressive learning jointly increases image size and regularization during training to further speed up training.
Loading 2104.00298v3…