PaperlayerView exact revision on arXiv

Source-linked AI summary

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Mingxing Tan, Quoc V. Le

arXiv:1905.11946v5cs.LGcs.CVstat.ML

TL;DR

ConvNet scaling across depth, width, and resolution remains poorly understood. This paper proposes compound scaling and EfficientNets, achieving higher accuracy and efficiency across ImageNet and transfer-learning benchmarks.

  • Problem

    ConvNet scaling lacks a well-understood approach for jointly choosing depth, width, and resolution under resource constraints.

  • Method

    The paper combines uniformly applied compound scaling with neural architecture search to create EfficientNets from an accuracy- and FLOPS-optimized baseline.

  • Results

    84.3% top1 accuracy is achieved by EfficientNet-B7, while EfficientNets generally use an order of magnitude fewer parameters and FLOPS than similarly accurate ConvNets.

  • Takeaways & Limitations

    Compound scaling improves accuracy over single-dimension scaling and supports efficient ConvNet scaling across ImageNet and transfer-learning datasets.

  • Takeaways & Limitations

    The evaluation omits ensemble and multi-crop models, as well as models pretrained on 3.5B Instagram images.

Abstract

from arXiv · show

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet. To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.

1. Introduction

The paper argues that ConvNet scaling is not well understood because prior methods typically scale only depth, width, or image resolution. It proposes compound scaling, which balances all three dimensions with fixed ratios, and applies it to create EfficientNets with strong accuracy and efficiency.

  • EfficientNet results: EfficientNet-B7 achieves 84.3% top-1 accuracy while being 8.4x smaller and 6.1x faster than GPipe.The authors first show scaling works on MobileNets and ResNet, then use neural architecture search to develop a baseline and scale it into the EfficientNet family.
  • Motivation: Prior ConvNet scaling commonly increases only depth, width, or image resolution, leaving the overall scaling process poorly understood.Examples include scaling ResNet by adding layers and scaling models by width or image resolution.
  • Compound scaling: Balanced scaling of network width, depth, and resolution can be achieved by multiplying each dimension by a constant ratio.For 2^N times more computational resources, depth, width, and image size are increased by α^N, β^N, and γ^N, respectively.
  • Compound scaling: Compound scaling is motivated by the need for larger images to receive greater receptive fields and more channels for fine-grained patterns.The paper relates this intuition to prior theoretical and empirical evidence connecting network width and depth.
  • Transfer results: EfficientNets achieve state-of-the-art accuracy on 5 out of 8 widely used datasets while reducing parameters by up to 21x versus existing ConvNets.This demonstrates transfer performance beyond ImageNet alongside improved parameter efficiency.

2. Related Work

Prior ConvNet progress has emphasized larger models for accuracy, while efficiency efforts have focused on compression, handcrafted mobile architectures, and neural architecture search. Existing scaling methods adjust depth, width, or input resolution, but applying efficient architecture-search techniques to larger models remains unclear.

  • ConvNet Accuracy: 84.3% top-1 validation accuracy was reached by GPipe, following growth from GoogleNet’s 74.8% with about 6.8M parameters to SENet’s 82.7% with 145M parameters.These results illustrate the field’s trend toward increasing ConvNet size for higher ImageNet accuracy.
  • ConvNet Efficiency: Model compression reduces ConvNet size by trading accuracy for efficiency, while SqueezeNets, MobileNets, and ShuffleNets are handcrafted mobile-size alternatives.These approaches address the overparameterization of deep ConvNets and the need for efficient models on mobile phones.
  • ConvNet Efficiency: Neural architecture search improves mobile ConvNet efficiency by tuning network width, depth, convolution kernel types, and kernel sizes.The passage identifies neural architecture search as increasingly popular for designing efficient mobile-size ConvNets.
  • ConvNet Efficiency: Applying these architecture-search techniques to larger models remains unclear because their design spaces and tuning costs are much larger.This limitation motivates studying how efficient-model techniques extend beyond mobile-size ConvNets.
  • Model Scaling: ResNet scales through depth, WideResNet and MobileNets through width, and larger input images can improve accuracy at the cost of more FLOPS.These examples represent distinct approaches to adapting ConvNets to different resource constraints.

3. Compound Model Scaling

Model scaling expands a fixed baseline network’s depth, width, and/or resolution, but scaling any single dimension eventually yields diminishing accuracy gains. The paper therefore proposes balancing all three dimensions with a compound coefficient that scales them uniformly and assigns resources according to their computational costs.

  • Scaling Problem: Model scaling expands a predefined baseline by changing network depth, width, and/or resolution without altering its layer operators.Restricting scaling to shared stage-level dimensions reduces the design space of possible configurations.
  • Single-Dimension Scaling: Scaling any single dimension improves accuracy, but gains diminish as models become larger.This limitation motivates moving beyond conventional single-dimension scaling.
  • Balanced Scaling: Balancing network width, depth, and resolution is critical for better accuracy and efficiency during ConvNet scaling.Width scaling saturates quickly at d=1.0 and r=1.0, while deeper and higher-resolution networks perform better at the same FLOPS cost.
  • Compound Scaling: The proposed compound scaling method uses a coefficient φ to uniformly scale network width, depth, and resolution in a principled way.Constants α, β, and γ determine how additional resources are assigned to width, depth, and resolution, respectively, using a small grid search.
  • Compound Scaling: Regular-convolution FLOPS scale proportionally with depth, width^2, and resolution^2, so the method constrains α · β^2 · γ^2 ≈ 2 to make FLOPS increase approximately 2^φ.Doubling depth doubles FLOPS, whereas doubling width or resolution increases FLOPS fourfold.

4. EfficientNet Architecture

EfficientNet-B0 is a mobile-size baseline designed through multi-objective neural architecture search and built with MBConv blocks enhanced by squeeze-and-excitation. The baseline is then scaled to EfficientNet-B1–B7 using compound coefficients searched once on B0 and reused across models.

  • Baseline design: EfficientNet-B0 is developed as a new mobile-size baseline using multi-objective neural architecture search that optimizes accuracy and FLOPS.The optimization goal is ACC(m)×[FLOPS(m)/T]^w, with w=-0.07 controlling the accuracy–FLOPS trade-off.
  • Baseline architecture: EfficientNet-B0 uses mobile inverted bottleneck MBConv as its main building block, augmented with squeeze-and-excitation optimization.Its FLOPS target is 400M, making the baseline slightly larger than the referenced network.
  • Compound scaling: The scaling search fixes φ = 1 and finds α = 1.2, β = 1.1, γ = 1.15 under the constraint α · β^2 · γ^2 ≈2.These coefficients are selected by a small grid search based on the compound-scaling equations.
  • Compound scaling: The fixed coefficients are then applied with different φ values to scale EfficientNet-B0 into EfficientNet-B1 to B7.This two-step procedure searches coefficients only on the small baseline and reuses them for the larger models.
  • Search efficiency: Searching scaling coefficients directly around a large model may improve performance, but its cost becomes prohibitively expensive.The proposed method avoids repeated large-model searches by performing the search once on EfficientNet-B0.

5. Experiments

Experiments show that compound scaling improves existing ConvNets and yields EfficientNets with substantially better accuracy-efficiency tradeoffs on ImageNet and transfer-learning datasets. EfficientNets also achieve strong real-hardware latency and parameter efficiency.

  • Scaling Existing ConvNets: Compound scaling improves ImageNet accuracy across MobileNets and ResNet compared with single-dimension scaling methods.This supports the scaling method's effectiveness on existing ConvNets.
  • Transfer Learning: 9.6x fewer parameters on average: EfficientNets achieve new state-of-the-art accuracy on 5 out of 8 transfer-learning datasets.Compared with publicly available models, they achieve 4.7x average and up to 21x parameter reduction.
  • ImageNet Results: 84.3% top1 accuracy: EfficientNet-B7 uses 66M parameters and 37B FLOPS, while being 8.4x smaller than GPipe.The reported gains are attributed to better architecture, scaling, and EfficientNet-specific training settings.
  • ImageNet Results: 18x fewer FLOPS: EfficientNet-B3 achieves higher accuracy than ResNeXt-101.Across representative ConvNets, scaled EfficientNets achieve better accuracy with fewer parameters and FLOPS.
  • Inference Latency: 5.7x faster: EfficientNet-B1 runs faster than ResNet-152, while EfficientNet-B7 runs about 6.1x faster than GPipe on a real CPU.Latency is averaged over 20 runs with batch size 1 on a single CPU core.

6. Discussion

The discussion separates the scaling method’s contribution from the EfficientNet architecture and finds that compound scaling improves accuracy more than single-dimension alternatives. Visualization suggests this advantage is associated with attention to more relevant regions and finer object details.

  • Scaling-method comparison: Figure 8 compares different scaling methods on the same EfficientNet-B0 baseline to disentangle the scaling method from the architecture.
  • Scaling-method comparison: Compound scaling improves accuracy by up to 2.5% more than other single-dimension scaling methods on the same EfficientNet-B0 baseline.All scaling methods improve accuracy at the cost of more FLOPS.
  • Activation-map analysis: Compound scaling tends to focus on more relevant regions with more object details than other scaling methods in class activation maps.The compared models are all scaled from the same baseline, using randomly selected ImageNet validation images.

7. Conclusion

The paper identifies balanced scaling of network width, depth, and resolution as essential for improving ConvNet accuracy and efficiency. It proposes a simple compound scaling method that scales baseline ConvNets to target resource constraints while maintaining efficiency.

  • 7. Conclusion: Balanced network width, depth, and resolution is identified as an important missing factor in ConvNet scaling for improving accuracy and efficiency.The paper systematically studies ConvNet scaling and highlights this balance as a central issue.
  • 7. Conclusion: The proposed compound scaling method uniformly enables principled scaling of a baseline ConvNet to target resource constraints.The method is described as simple and highly effective.
  • 7. Conclusion: The compound scaling method scales baseline ConvNets while maintaining model efficiency.It is intended to address the accuracy–efficiency limitation in existing scaling practices.

Appendix

The paper primarily reports ImageNet validation accuracy for comparability, while also verifying test accuracy on the 100k-image test set. Test accuracy is reported as close to validation accuracy in Table 8.

  • ImageNet Accuracy Verification: ImageNet validation accuracy is used as the primary comparison convention, consistent with most research papers since 2017.The paper also verifies test accuracy by submitting predictions on the 100k test images.
  • ImageNet Accuracy Verification: Test accuracy is very close to validation accuracy for ImageNet top-1 and top-5 measures.Table 8 compares ImageNet validation versus test top-1/5 accuracy.
Loading 1905.11946v5…