PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets

Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, Tong Zhang

arXiv:2010.14819v2cs.CV

TL;DR

Rules for shrinking neural networks by jointly adjusting resolution, depth, and width remain underexplored, and EfficientNet's compound scaling is designed for enlarging models. This paper proposes a tiny formula that prioritizes resolution and depth and sets width under a FLOPs constraint. TinyNet models generally outperform competing small CNNs, with TinyNet-E reaching 59.9% Top-1 accuracy at 24M FLOPs.

  • Problem

    EfficientNet's compound scaling addresses network enlargement, while effective rules for shrinking models by adjusting resolution, depth, and width remain underexplored.

    Source: §1 · p. 2

  • Method

    The tiny formula estimates resolution and depth from frontier models using Gaussian process regression, then determines width to satisfy the FLOPs constraint.

    Source: §1 · p. 2

  • Results

    TinyNet-E achieves 59.9% Top-1 accuracy with 24M FLOPs, 1.9% higher than MobileNetV3 Small 0.5× at similar computational cost.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The tiny formula produces small, effective networks on both EfficientNet and ResNet architectures.

    Source: §5 · p. 9

Abstract

from arXiv · show

To obtain excellent deep neural architectures, a series of techniques are carefully designed in EfficientNets. The giant formula for simultaneously enlarging the resolution, depth and width provides us a Rubik's cube for neural networks. So that we can find networks with high efficiency and excellent performance by twisting the three dimensions. This paper aims to explore the twisting rules for obtaining deep neural networks with minimum model sizes and computational costs. Different from the network enlarging, we observe that resolution and depth are more important than width for tiny networks. Therefore, the original method, i.e., the compound scaling in EfficientNet is no longer suitable. To this end, we summarize a tiny formula for downsizing neural architectures through a series of smaller models derived from the EfficientNet-B0 with the FLOPs constraint. Experimental results on the ImageNet benchmark illustrate that our TinyNet performs much better than the smaller version of EfficientNets using the inversed giant formula. For instance, our TinyNet-E achieves a 59.9% Top-1 accuracy with only 24M FLOPs, which is about 1.9% higher than that of the previous best MobileNetV3 with similar computational cost. Code will be available at https://github.com/huawei-noah/ghostnet/tree/master/tinynet_pytorch, and https://gitee.com/mindspore/mindspore/tree/master/model_zoo/research/cv/tinynet.

1 Introduction

EfficientNet's compound scaling enlarges resolution, depth, and width, but rules for shrinking a baseline remained underexplored. The paper proposes a tiny formula for downsizing and reports TinyNet models with strong accuracy at lower computational cost.

1 Introduction

  • Mobile deployment motivates adjusting network depth, width, and image resolution to reduce memory and latency, while EfficientNet scales all three dimensions together.Source: §1 · p. 1
  • EfficientNet and improved variants focus on scaling up, leaving rules for effectively downsizing a baseline model insufficiently explored.Source: §1 · p. 2
  • Among 100 randomly generated downsized models, the best perform about 2.5% better than models from EfficientNet's inverse giant formula across different FLOPs.Figure 1 highlights inverse-formula models in green and frontier models with both higher performance and lower FLOPs in red.Source: §1 · p. 2, Figure 1 · p. 2
  • TinyNet-E reaches 59.9% Top-1 accuracy with 24M FLOPs, 1.9% higher than MobileNetV3 Small 0.5× at similar computational cost.TinyNet-A achieves 76.8% Top-1 accuracy with about 339M FLOPs, while EfficientNet-B0 needs about 387M FLOPs for similar performance.Source: §1 · p. 2
  • The tiny formula prioritizes resolution and depth, estimates their values from frontier models with Gaussian process regression, and sets width to meet the FLOPs constraint.The paper also states that the formula can be applied to ResNet architectures.Source: §1 · p. 2

2 Related Work

Prior work reduces model costs through compression techniques and studies depth, width, and resolution as ways to trade computation for performance. How to shrink a given model into a compact version remains open.

2 Related Work

  • Compression methods target lower computation, energy, and storage costs through pruning, low-bit quantization, low-rank factorization, and knowledge distillation.Source: §2 · p. 2
  • Compression methods are usually bounded by the performance of their pretrained models, while low-rank factorization can require computationally expensive decomposition.Source: §2 · p. 3
  • Prior work varies network depth, width, or input resolution, and EfficientNet combines all three for scaling up; shrinking a given model remains an open problem.ResNet offers different depths, WideResNet explores wider and shallower residual networks, and higher-resolution inputs provide more information at greater computational cost.Source: §2 · p. 3

3 Approach

The approach examines how resolution, depth, and width affect smaller networks, then uses those relationships to derive a FLOPs-constrained shrinking formula. Its Pareto-front analysis finds the strongest correlation with FLOPs for resolution, followed by depth and width.

3.1 Rethinking the Importance of (r, d, w)

  • At roughly 0.5C0 FLOPs on ImageNet-100, accuracy tracks resolution most; top accuracies cluster at r=0.8–1.4, while strong models span d=0.5–2 and mostly use w<1.At fixed FLOPs, width has a roughly negative correlation with accuracy.Source: Figure 2 · p. 3, §3.1 · pp. 3–4
  • EfficientNet-B−1 uses r=0.86, d=0.8, and w=0.89 at 0.5C0 FLOPs, but its 75.8% ImageNet-100 accuracy is below that of some randomly generated models.This motivates searching for a shrinking rule rather than simply inverting EfficientNet’s compound scaling.Source: §3.1 · p. 4

3.2 Tiny Formula for Model Rubik’s Cube

  • On the Accuracy-FLOPs Pareto front, resolution correlates most with FLOPs (Spearmanr 0.81), followed by depth and width.The analysis selects the top 20% of models by performance and computational complexity using NSGA-III nondominated sorting.Source: Figure 3 · p. 4, §3.2 · p. 4
  • The tiny formula models resolution and depth against the FLOPs reduction factor using Gaussian process regression, then determines width from the FLOPs constraint.For TinyNet, the procedure starts with EfficientNet-B0 and uses an RBF covariance kernel to generate smaller models at specified FLOPs levels.Source: §3.2 · pp. 4–5
  • The shrinking rule is based on frontier small models rather than EfficientNet’s handcrafted compound-scaling rule.The paper presents it as a way to produce tiny networks with higher performance.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate model shrinking on ImageNet-100 and ImageNet-1000, comparing the proposed TinyNet formula with EfficientNet-based and other alternatives. TinyNet performs better than competing small models, with additional results on training behavior, inference latency, and object detection.

4.1 Datasets and Experimental Settings

  • ImageNet-1000 evaluation uses 1.2 million training and 50,000 validation images across 1,000 classes; ImageNet-100 uses 500 training images per class and 5,000 validation images.EfficientNet-B0-based models train for 450 epochs; inverse-rule baselines B−1 through B−4 have about 200M, 100M, 50M, and 25M FLOPs, respectively.Source: §4.1 · pp. 5–6

4.2 Experiments on ImageNet-100

  • The proposed tiny formula outperforms EfficientNet’s inverse rule and separate resolution-, depth-, or width-only shrinking on ImageNet-100.The compared models are shrunken from the EfficientNet-B0 baseline.Source: Table 1 · p. 6, §4.2 · p. 6
  • On ImageNet-100, shrunken ResNet-50 models generally outperform alternatives based on layer reduction or the EfficientNet rule.The models are compared on ImageNet-100.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Experiments on ImageNet-1000

  • 59.9% Top-1 accuracy at 24M FLOPs gives TinyNet-E a 1.9% lead over MobileNetV3 Small 0.5× at similar computational cost.Separately, TinyNet-A + RA reaches 77.7% Top-1 accuracy, 0.9% above vanilla TinyNet-A.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • By the end of training, TinyNet-E outperforms EfficientNet-B−4 by 3.2% accuracy.Class activation maps show TinyNet-E attending to more relevant regions, while EfficientNet-B−4 sometimes focuses on unrelated objects or local target parts.Source: Figure 4 · p. 7, §4.3 · pp. 7–8, Figure 5 · p. 8
  • As FLOPs decrease, accuracy falls across shrinking approaches, but TinyNet alleviates the drop; B0’s minimum viable depth is seven bottlenecks at 174M FLOPs.The comparison includes shrinking by resolution, depth, or width separately.Source: Figure 6 · p. 8, §4.3 · p. 8
  • TinyNet-A runs 15% faster than EfficientNet-B0 with similar accuracy, while TinyNet-E gains 3.2% accuracy over EfficientNet-B−4 at similar latency.Latency was measured on a Huawei P40 in single-threaded mode with batch size 1, averaging 1,000 runs.Source: Table 4 · p. 8, §4.3 · p. 8
  • On MS COCO object detection, TinyNet-D outperforms EfficientNet-B−3 by a large margin at comparable computational cost.The evaluation uses SSDLite with 512×512 input.Source: Table 5 · p. 8, §4.3 · p. 8

5 Conclusion and Discussion

The conclusion finds that EfficientNet’s giant formula is unsuitable for shrinking networks and proposes focusing on resolution and depth while calculating width under a FLOPs constraint. TinyNets and experiments on EfficientNets and ResNets support the scheme, while further enlargement remains future work.

5 Conclusion and Discussion

  • The tiny formula focuses on resolution and depth, then calculates width to satisfy the FLOPs constraint.It is based on observations about smaller models and is used to twist the three dimensions into a series of TinyNets.Source: §5 · p. 9
  • Experiments on EfficientNets and ResNets demonstrate the effectiveness of the proposed scheme for designing tiny networks.Source: §5 · p. 9
  • The tiny formula summarizes observations on smaller models; enlarging those models with new rules beyond EfficientNet’s giant formula is left for future work.Source: §5 · p. 9

Broader Impact

The paper frames model shrinking as beneficial for energy conservation and environmental protection amid the resource demands of widely used deep networks.

Broader Impact

  • Deep neural networks require substantial computation resources, putting pressure on energy sources and the natural environment.Source: Broader Impact · p. 9
  • The authors state that their model-shrinking method for obtaining tiny neural networks is beneficial to energy conservation and environmental protection.Source: Broader Impact · p. 9

A.1 Experiments on GhostNet

The proposed shrinking approach produces smaller GhostNet models and compares them with width-only GhostNets and other ImageNet models. The enhanced GhostNets outperform the compared models, while activation functions and data augmentation can further boost performance.

A.1 Experiments on GhostNet

  • The smaller GhostNet-B/C/D models obtained with the tiny formula outperform original GhostNets that only change width.The models are derived by shrinking GhostNet-A, a baseline with about 591M FLOPs.Source: §A.1 · p. 9, Table 6 · p. 10
  • More complex activations such as HSwish and RandAugment can boost GhostNet performance beyond the ReLU setting.Source: §A.1 · p. 9, Table 6 · p. 10
  • GhostNet models enhanced by the TinyNet technique consistently outperform compared state-of-the-art models by a significant margin.Figure 7 plots Top-1 accuracy versus FLOPs on ImageNet.Source: §A.1 · p. 9, Figure 7 · p. 10

Related papers

Loading 2010.14819v2…