PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientNetV2: Smaller Models and Faster Training

Mingxing Tan, Quoc V. Le

arXiv:2104.00298v3cs.CV

TL;DR

As models and datasets grow, training can demand substantial resources. EfficientNetV2 combines training-aware search and scaling with adaptive progressive learning; across ImageNet, CIFAR, Cars, and Flowers, it trains up to 11x faster and is up to 6.8x more parameter-efficient than prior models.

  • Problem

    As models and datasets grow, training efficiency matters because large-scale training can require weeks and thousands of GPUs, making retraining or improvement difficult.

    Source: §1 · p. 1

  • Method

    The paper combines training-aware neural architecture search and scaling to optimize accuracy, training speed, and parameter efficiency, while progressively adjusting image size and regularization.

    Source: §1 · p. 2

  • Results

    Across ImageNet, CIFAR, Cars, and Flowers, EfficientNetV2 trains up to 11x faster and is up to 6.8x more parameter-efficient than prior models.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    With 32 TPUv3 cores, EfficientNetV2 can complete ImageNet21k training within two days, according to the authors.

    Source: §1 · p. 2

    The proposed explanation for accuracy drops across image sizes—that fixed regularization is unbalanced—is framed as a hypothesis.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.

1. Introduction

EfficientNetV2 combines training-aware architecture search and scaling with progressive learning that adjusts regularization as image size grows, targeting faster training and better parameter efficiency. The introduction reports strong benchmark accuracy alongside substantial speed and size gains.

1. Introduction

  • Training efficiency matters as models and datasets grow, while prior approaches can incur expensive parameter-size overhead.The introduction notes that GPT-3 required weeks of training on thousands of GPUs, making retraining or improvement difficult.Source: §1 · p. 1
  • 87.3% top-1 accuracy on ImageNet ILSVRC2012 after ImageNet21k pretraining exceeds ViT-L/16 by 2.0% while training 5x–11x faster.The paper also reports 85.7% top-1 accuracy on ImageNet while training 3x–9x faster and being up to 6.8x smaller; results span ImageNet, CIFAR, Cars, and Flowers.Source: Figure 1 · p. 1, §1 · p. 2
  • Training-aware NAS and scaling jointly optimize accuracy, training speed, and parameter size using a search space enriched with Fused-MBConv.The search was motivated by EfficientNet bottlenecks: very large images train slowly, depthwise convolutions are slow in early layers, and scaling every stage equally is suboptimal.Source: §1 · pp. 1–2
  • Progressive learning starts with smaller images and weaker regularization, then increases both to speed training without an accuracy drop.The approach addresses the mismatch between image size and regularization: smaller images call for weaker regularization, while larger images need stronger regularization.Source: §1 · p. 2

2. Related work

Related work often targets parameter efficiency, training or inference speed, or progressive training separately. This paper distinguishes its approach by using NAS to optimize training speed and parameter efficiency together and by adaptively adjusting regularization during progressive training.

Training and parameter efficiency

  • Prior work often targets parameter efficiency or training and inference speed separately, whereas this paper uses NAS to jointly optimize training speed and parameter efficiency.Speed-focused models often use more parameters; the paper’s goal is to improve both dimensions.Source: §2 · pp. 2–3

Progressive training

  • Progressive resizing and Mix&Match use fixed regularization across image sizes, while this approach adapts regularization to improve training speed and accuracy.It also gradually increases learning difficulty by adding regularization, rather than selecting training examples from easy to hard.Source: §2 · p. 2

Neural architecture search

  • Earlier NAS work often optimizes FLOPs efficiency or inference efficiency, while this paper uses NAS to optimize training and parameter efficiency.NAS has also been applied to image classification, detection, segmentation, and hyperparameter optimization.Source: §2 · pp. 2–3

3. EfficientNetV2 Architecture Design

The section identifies EfficientNet training bottlenecks and introduces training-aware neural architecture search and scaling to improve training speed while maintaining parameter efficiency.

3.1. Review of EfficientNet

  • EfficientNet uses NAS to find a FLOPs-efficient B0, then compound-scales it into B1–B7; this section aims to improve training speed while maintaining parameter efficiency.The paper notes that recent methods claiming speed gains are often less efficient in parameters and FLOPs than EfficientNet.Source: §3.1 · p. 3

3.2. Understanding Training Efficiency

  • Training with smaller images reduces computation and enables larger batches, improving training speed by up to 2.2x; the authors use no post-training layer fine-tuning.Smaller training images are also reported to yield slightly better accuracy; the passage contrasts this with FixRes, which fine-tunes after training.Source: §3.2 · p. 3, Table 2 · p. 3
  • Replacing MBConv with Fused-MBConv in early stages can speed training with small parameter and FLOPs overhead, but replacing every stage increases both and slows training.Fused-MBConv replaces MBConv’s depthwise 3x3 and expansion 1x1 convolutions with a regular 3x3 convolution; choosing the right mixture motivates NAS.Source: §3.2 · p. 3, Table 3 · p. 3, Figure 2 · p. 3
  • EfficientNetV2 uses non-uniform scaling that adds layers to later stages and caps maximum image size to address unequal stage contributions and image-related memory costs.The section identifies equal stage scaling as sub-optimal and aggressive image-size scaling as a source of high memory use and slow training.Source: §3.2 · pp. 3–4

3.3. Training-Aware NAS and Scaling

  • The training-aware NAS jointly optimizes accuracy, parameter efficiency, and accelerator training efficiency over operation type, depth, kernel size, and expansion ratio choices.It samples up to 1,000 models, trains each for about 10 epochs at reduced image size, and scores them with A · S^w · P^v, where w = -0.07 and v = -0.05.Source: §3.3 · p. 4
  • EfficientNetV2-S combines MBConv and Fused-MBConv early, favors smaller MBConv expansion ratios and 3x3 kernels with more layers, and removes the final stride-1 stage.The passage attributes these choices to memory-access overhead, receptive-field compensation, and the large parameter size and memory-access overhead of the removed stage.Source: §3.3 · p. 4, Table 4 · p. 4
  • With fixed image sizes and similar training-step counts, EfficientNetV2 trains much faster than recent models, while properly trained EfficientNets retain a strong performance trade-off.The comparison uses 350 epochs for most models and 360 for NFNets; EfficientNet is also shown with a roughly 30% smaller training image size.Source: Figure 3 · p. 4, §3.3 · p. 4

4. Progressive Learning

Progressive learning addresses accuracy losses from changing image sizes by adapting regularization to image size. It begins with smaller images and weaker regularization, then increases both through training.

4.1. Motivation

  • Changing image size during training can reduce accuracy; the authors hypothesize that this reflects fixed regularization mismatched to image-dependent model capacity.Smaller images imply lower capacity and call for weaker regularization, while larger images imply greater capacity and vulnerability to overfitting.Source: §4.1 · p. 5
  • Small images achieved their best accuracy with weak augmentation, whereas larger images performed better with stronger augmentation, motivating adaptive regularization.The comparison used a model sampled from the search space and varied image sizes and data augmentations.Source: §4.1 · p. 5, Table 5 · p. 5

4.2. Progressive Learning with adaptive Regularization

  • Progressive learning starts with small images and weak regularization, then gradually increases image size and regularization to make training more difficult.The approach adapts dropout, RandAugment magnitude, and mixup ratio as training progresses.Source: §4.2 · p. 5, Figure 4 · p. 5
  • Training is divided into stages with interpolated image sizes and regularization, ending at target settings; each stage inherits the previous stage’s weights.The initial image size and regularization are chosen heuristically, with linear interpolation used to set stage values.Source: §4.2 · p. 5

5. Main Results

The section reports ImageNet results, ImageNet21k pretraining results, and transfer-learning results on CIFAR-10, CIFAR-100, Cars, and Flowers.

5.1. ImageNet ILSVRC2012

  • EfficientNetV2-L achieves 85.7% top-1 accuracy, surpassing the larger ViT-L/16(21k) pretrained on ImageNet21k.The passage notes that ViTs are not well tuned on ILSVRC2012, while DeiTs improve results with more regularization.Source: §5.1 · p. 6

5.2. ImageNet21k

  • Beyond 85% top-1 accuracy, scaling data is more effective than simply scaling model size; ImageNet21k pretraining can improve accuracy despite overfitting from model-size scaling.The authors also report that EfficientNetV2 pretraining on ImageNet21k takes under two days using 32 TPU cores, versus weeks for ViT.Source: §5.2 · pp. 6, 8

5.1. ImageNet ILSVRC2012

  • EfficientNetV2-M matches EfficientNet-B7 accuracy while training 11x faster; EfficientNetV2 models also outperform recent RegNet and ResNeSt in accuracy and inference speed.The authors attribute the speedup to both progressive training and improved networks.Source: §5.1 · p. 6, Table 7 · p. 7

5.2. ImageNet21k

  • EfficientNetV2-L(21k) reaches 86.8% top-1 accuracy versus ViT-L/16(21k) at 85.3%, with 2.5x fewer parameters, 3.6x fewer FLOPs, and 6x–7x faster training and inference.The models tagged 21k are pretrained on ImageNet21k and finetuned on ILSVRC2012.Source: §5.2 · p. 6, Table 7 · p. 7

5.1. ImageNet ILSVRC2012

  • EfficientNetV2 inference latency is up to 3x faster than EfficientNets; EfficientNetV2-M is 0.6% more accurate and 2.8x faster at inference than GPU-optimized ResNeSt.Inference latency comparisons use the same PyTorch Image Models codebase and machine, with batch size 16.Source: §5.1 · p. 6, Figure 5 · p. 7

5.3. Transfer Learning Datasets

  • On CIFAR-100, EfficientNetV2-L achieves 0.6% better accuracy than prior GPipe/EfficientNets and 1.5% better accuracy than prior ViT/DeiT models; overall, our models outperform previous ConvNets and Vision Transformers on the compared datasets.Table 8 compares models pretrained on ImageNet ILSVRC2012 and finetuned on downstream datasets; accuracy is averaged over five runs.Source: §5.3 · p. 8, Table 8 · p. 8

6. Ablation Studies

EfficientNetV2 outperforms EfficientNet under matched training settings, progressive learning improves efficiency across ResNets and EfficientNets, and adaptive regularization boosts progressive resizing results.

6.1. Comparison to EfficientNet

  • EfficientNetV2-M uses 17% fewer parameters and 37% fewer FLOPs than EfficientNet-B7, while training 4.1x faster and inferring 3.1x faster.Both models use the same training settings, so the paper attributes these gains to the EfficientNetV2 architecture.Source: Table 10 · p. 8, §6.1 · p. 8
  • Without progressive learning, scaled-down EfficientNetV2-S models are generally faster than small EfficientNets while maintaining comparable parameter efficiency.The paper presents adaptive regularization as a key insight that adjusts regularization with image size; its simple progressive approach can also be combined with other methods.Source: §6.1 · p. 8, Table 11 · p. 9, §6.3 · p. 9

6.2. Progressive Learning for Different Networks

  • Progressive learning generally reduces training time and improves accuracy across networks, with 2.1x speedup for ResNet152(380) and 2.2x for EfficientNet-B4.The speedup is limited to 1.4x for ResNet50(224), whose default image size is already small.Source: Table 12 · p. 9, §6.2 · p. 9

6.3. Importance of Adaptive Regularization

  • Adaptive regularization improves accuracy by 0.7% over vanilla progressive or random resizing with fixed regularization, and smaller regularization on early small images speeds convergence.For random resizing, image size is sampled every eight epochs rather than every batch because TPU requires recompilation for each new size.Source: Table 13 · p. 9, §6.3 · p. 9, Figure 6 · p. 9

7. Conclusion

EfficientNetV2 is a family of smaller, faster image-recognition networks optimized with training-aware NAS and model scaling. The paper also proposes progressive learning that jointly increases image size and regularization during training.

7. Conclusion

  • Compared with EfficientNet and more recent works, EfficientNetV2 trains up to 11x faster and is up to 6.8x smaller.Experiments report strong results on ImageNet, CIFAR, Flowers, and Cars.Source: §7 · p. 9
  • Training-aware NAS and model scaling optimize EfficientNetV2 for training speed and parameter efficiency.The paper presents EfficientNetV2 as a new family of smaller, faster neural networks for image recognition.Source: §7 · p. 9
  • Progressive learning jointly increases image size and regularization during training to further speed up training.Source: §7 · p. 9

Related papers

Loading 2104.00298v3…