Source-linked AI summary
Fast and Accurate Model Scaling
Piotr Dollár, Mannat Singh, Ross Girshick
TL;DR
Existing scaling analyses often focus on accuracy versus FLOPs, although scaled models with equal FLOPs can have very different runtimes. This paper analyzes model properties related to runtime and introduces fast scaling, which emphasizes width. Fast scaling yields fast, accurate models, including a 16GF RegNetY-4GF model that is faster and more accurate than EfficientNet-B4 despite the latter having 4× fewer FLOPs.
Problem
Existing scaling work often focuses on accuracy versus FLOPs, despite evidence that equal-FLOP models can have very different runtimes on modern accelerators.
Source: §1 · p. 2Method
The paper analyzes scaling in terms of model properties including activations and introduces a one-parameter fast scaling strategy that primarily scales width.
Source: §1 · p. 2Results
A 16GF RegNetY-4GF scaled with fast scaling is faster, more accurate, and uses less memory than EfficientNet-B4, which has 4× fewer FLOPs.
Source: §1 · p. 2Takeaways & Limitations
The framework supports reasoning about model scaling beyond FLOPs, while the optimal scaling approach may depend on the task.
Source: §7 · p. 8Uniformly scaling model stages can leave models with more parameters than optimized alternatives, which may use fewer blocks in the last stage at higher FLOP regimes.
Source: §6.4 · p. 8
Abstract
from arXiv · showhide
In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about $O(s)$ increase in model activation w.r.t. scaling flops by a factor of $s$, the proposed fast compound scaling results in close to $O(\sqrt{s})$ increase in activations, while achieving excellent accuracy. This leads to comparable speedups on modern memory-limited hardware (e.g., GPU, TPU). More generally, we hope this work provides a framework for analyzing and selecting scaling strategies under various computational constraints.
1. Introduction
In high-compute settings, scaling must balance accuracy with practical runtime, yet equal-FLOP models can run very differently. The paper analyzes these tradeoffs and proposes fast scaling, which emphasizes width while controlling activation growth.
1. Introduction
- Multiple scaling strategies can produce similarly accurate models at the same FLOPs, but their runtimes can differ widely.This motivates considering runtime alongside accuracy and FLOPs when choosing how to scale.Source: §1 · p. 2
- Width scaling raises activations by O(√s), while compound and depth-width scaling approach O(s); fast scaling also approaches O(√s) and runs nearly as fast as width scaling.Across the strategies examined, activations correlate more strongly with runtime than FLOPs within a practical range on modern accelerators, making them a runtime proxy.Source: §1 · pp. 1–2
- Fast compound scaling uses a single parameter α to control width relative to other dimensions, enabling control of activation growth while producing fast, accurate models.The paper describes this strategy as primarily, but not exclusively, scaling model width.Source: §1 · p. 2
- Fast scaling makes a 16GF RegNetY-4GF more accurate, faster, and less memory-intensive than EfficientNet-B4, which has 4× fewer FLOPs.The authors present this as a concrete example of a large model matching or exceeding state-of-the-art accuracy while running faster.Source: §1 · p. 2
2. Related Work
Prior work expanded networks through manual design, architecture search, and reusable design spaces, while model scaling becomes important when extensive model comparison is infeasible. Existing scaling approaches include depth, width, resolution, and compound scaling.
Manual network design
- Manual design advanced through deeper architectures such as VGG and Inception, then residual networks; wider models and design elements such as depthwise convolution also enabled larger models.The cited history includes AlexNet, VGG [26], Inception [28] [29], residual networks [9], wider models [34] [11], depthwise convolution [3], group convolution [32], inverted bottlenecks [25], SE [12], and newer nonlinearities [10] [23].Source: §2 · p. 2
Automated network design
- Architecture search has produced efficient models, especially in low-compute settings, but is computationally expensive for larger models and most effective in low- or medium-compute regimes.DARTS [17] is cited as a differentiable search strategy that avoids training multiple separate models to optimize structure.Source: §2 · p. 2
Design space design
Network scaling
Going bigger
Network scaling
- Table 1 compares scaling one model dimension at a time and notes that parameter and activation costs vary substantially, especially at large scale.The table caption describes scaling a stage with d convolutional layers, width w, and spatial resolution r×r by a factor s.Source: Table 1 · p. 3
3. Complexity of Scaled Models
The section develops a framework for comparing scaled networks by flops, parameters, and activations, then shows how scaling choices change these measures. Activations grow more slowly under width scaling than under uniform compound scaling, while group-convolution behavior depends on how channel and group widths are scaled.
3.1. Complexity Metrics
- Flops, parameters, and activations measure multiply-adds, free variables, and convolution-output elements; activations matter for speed on memory-bandwidth-limited hardware.Parameters are independent of input resolution, so they do not fully reflect convolutional-network capacity or runtime.Source: §3.1 · p. 3
3.2. Network Complexity
- The framework approximates whole-network scaling through convolutional layers, assuming other layers’ counts and complexity track the convolutional layers.For a stage of d uniform layers, f = dw^2r^2, p = dw^2, and a = dwr^2; kernel size k is held fixed at 1 for analysis.Source: §3.2 · p. 3
3.4. Complexity of Compound Scaling
- Uniform compound scaling allocates flop growth across dimensions, but its activations increase nearly linearly with s.For dwr scaling, depth scales by the cube root of s and width and resolution by the sixth root of s.Source: Table 2 · p. 3, §3.4 · p. 3
3.3. Complexity of Simple Scaling
- For a stage scaled by factor s, activations grow by √s with width scaling but by s with depth or resolution scaling.Flops increase by s in each case, while parameters and activations vary by strategy.Source: §3.3 · p. 3
3.5. Complexity of Group Width Scaling
- For group convolutions, scaling channel width and group width together is needed for behavior similar to regular-convolution scaling; depthwise convolutions keep g = 1.The analysis scales g proportionally to w unless otherwise noted; joint width and group-width scaling increases activations by √s.Source: §3.5 · p. 4, §3.4 · p. 4, Table 3 · p. 4
4. Runtime of Scaled Models
The section compares flops, parameters, and activations as predictors of model runtime across scaling strategies. Activations are the strongest within-model proxy, though their predictive power has limits across different models and for very large models.
4. Runtime of Scaled Models
- r ≥0.99 for runtime versus activations across a single model’s scaling strategies, compared with 0.81 for flops and 0.56 for parameters.Figure 2 plots epoch time against each complexity metric; activations remain strongly predictive across the tested strategies.Source: §4 · p. 4, Figure 2 · p. 4
- r = 0.95 for runtime versus activations across different models; very large models can be flop-bound, and differing operations can affect runtime.The authors caution that activations do not perfectly predict runtime for heterogeneous models.Source: §4 · p. 4
- Runtime measurements use one ImageNet training epoch, the largest batch fitting in memory, PyTorch, and eight 32GB Volta GPUs.The authors note runtime depends on hardware and inference performance depends on the use case.Source: §4 · p. 4
- α = 0.8 is reported to give an excellent speed–accuracy tradeoff in the fast-scaling family, which scales primarily along width.The family is parameterized by α; α = 1 corresponds to width scaling and α = 1/3 to uniform compound scaling.Source: Table 4 · p. 5
5. Fast Compound Model Scaling
Fast compound scaling prioritizes width while increasing depth and resolution less, aiming to limit activation growth without giving up accuracy. Its scaling family is parameterized by α, with values near 1 designated fast scaling.
5. Fast Compound Model Scaling
- Increasing channel width and group width produces the smallest activation increase among strategies scaling width, depth, and resolution.Wide networks are also described as efficient in wall-clock time, though they may not always match deeper or higher-resolution models on top results.Source: §5 · p. 5
- Fast compound scaling primarily increases model width while increasing depth and resolution to a lesser extent.The family is parameterized by α, and for group convolutions group width is scaled in line with width.Source: §5 · p. 5
- α = 1 gives width scaling, α = 0 gives depth-and-resolution scaling, and α = 1/3 gives uniform compound scaling.The scaled model’s complexity is expressed in terms of depth, width, and resolution.Source: §5 · p. 5
- Fast scaling uses 1/3 < α < 1, with α = 0.8 as the default, denoted dWr.The paper reports that this regime yields good speed and accuracy.Source: §5 · p. 5
6. Experiments
The experiments compare scaling strategies across EfficientNet and RegNet families, then test fast scaling, search, and larger models. Across these settings, accuracy, activations, and runtime vary with the scaling choice, while fast scaling aims to balance accuracy and speed.
6. Experiments
- The experiments assess existing scaling strategies, fast scaling, scaling versus random search, and large models across the three baseline families.RegNets also permit comparisons between scaled models and models selected by random search.Source: §6 · p. 5, §6.1 · p. 5
6.1. Baselines and Optimization Settings
- The study evaluates EfficientNet, RegNetY, and RegNetZ with a shared reproducible optimization setup; RegNetZ adds an inverted-bottleneck block, and the 4× EfficientNet schedule outperforms originally reported results except for the largest tested model.The 2× schedule is used in following experiments unless otherwise noted.Source: §6.1 · pp. 5–6, Table 5 · p. 6
6.2. Simple and Compound Scaling
- Uniform compound scaling achieves the best accuracy for EfficientNet and RegNet, but is slower than width scaling; uniform compound scaling also closely matches original EfficientNet compound scaling.For RegNets, group width is scaled proportionally to width.Source: §6.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 7
6.3. Fast Scaling
- With α < 1, EfficientNet scaling achieves substantially higher accuracy than width-only scaling, while dWr (α = 4/5) is nearly as fast as width scaling; RegNet results are similar.For RegNets, scaling group width alongside width helps explain why width scaling is more effective than it is for EfficientNet.Source: §6.3 · p. 7
6.4. Scaling versus Search
- A 4GF→16GF hybrid RegNetY is 0.3% better than scaling a 500MF model directly to 16GF, although searched 4GF models outperform directly scaled models.For RegNetY and RegNetZ at 4GF, the best sampled models outperform scaled models by 0.6% and 0.1%, respectively.Source: §6.4 · p. 8, Table 7 · p. 8
6.5. Comparison of Large Models
- At 16GF, the error gap between RegNetY and RegNetZ shrinks to 0.8% from 2.2% at 500MF; fast-scaled RegNetY and RegNetZ are asymptotically faster than original EfficientNets.RegNetY is the fastest model tested, while scaled RegNetZ achieves the best accuracy; RegNetY-4GF→16GF uses less memory and is faster than EfficientNet-B4 despite having ∼4× as many flops.Source: §6.5 · p. 8, Figure 8 · p. 8
7. Discussion
The paper presents a framework for analyzing scaling beyond flops and proposes fast scaling that primarily increases width. It reports accurate, fast models while noting that the best approach may depend on the task.
7. Discussion
- Fast scaling primarily, but not exclusively, increases width and yields accurate models with fast runtime; the framework considers activations, which are highly correlated with runtime on modern hardware.The appendix notes that group convolution may be a better option at higher compute regimes, while model speed rankings can depend on implementation.Source: §7 · p. 8, Figure 9 · p. 9, Figure 10 · p. 9
Appendix
The appendix reports that activations closely track runtime across RegNet scaling strategies and discusses implementation-dependent speed comparisons. It also examines the role of group convolution in RegNetZ’s performance relative to EfficientNet.
Large models additional analysis
- RegNetY and RegNetZ offer strong speed–accuracy tradeoffs, but their relative speed ranking may change with implementation optimizations.The passage notes that group convolution may be underoptimized relative to depthwise or full-width convolution.Source: Appendix · p. 9
Group vs. depthwise conv
- RegNetZ-G1, which substitutes depthwise for group convolution, achieves better accuracy than EfficientNet, while group convolution accounts for much of RegNetZ’s improvement at higher flops.Group-convolution speed rankings may change with better implementations; activations remain strongly predictive of runtime.Source: Appendix · p. 9
RegNet timing results
- Across RegNetY and RegNetZ scaling strategies, runtime plots closely follow activation plots; width scaling or larger α gives the lowest activations and runtime.Figure 11 shows width scaling results; Figure 12 shows large α results.Source: Appendix · p. 9, Figure 11 · p. 9, Figure 12 · p. 9