Measuring the Algorithmic Efficiency of Neural Networks

1 Introduction

• AI progress is important to policymakers, economists, industry leaders, and researchers, yet hardware trends are easier to quantify than algorithmic improvement.The authors seek measures that isolate algorithmic improvement to help paint a picture of overall field progress.Source: §1.1 · p. 3
• The paper measures algorithmic progress by holding performance constant and comparing training compute, rather than inference cost or parameter counts.The authors focus on training because they are more interested in what systems are possible to produce than the cost of running them; they measure total FLOPs because research is typically FLOPs-bound.Source: §1.1 · pp. 3–4
• Different performance units and incomparable task difficulties make aggregate progress hard to interpret, while performance-focused reporting obscures efficiency gains from compute increases.Benchmarks are also being solved increasingly rapidly, leaving less time to assess progress.Source: §1.1 · p. 3
• 79.1% top-5 accuracy on ImageNet defines the paper’s AlexNet-level target, chosen because AlexNet helped spark neural-network interest and ImageNet remains a widely followed benchmark.The paper presents its efficiency measurements as useful both to decision makers and to measuring contributions made with less compute.Source: §1.1 · pp. 3–4

2.1 Algorithmic progress had similar rate to Moore’s Law in some domains over decades

• Algorithmic gains in several domains were estimated at roughly 50–100% of hardware gains, but the estimates relied on substantial judgment and may be optimistic.The reviewed work covered chess, Go, physics simulations, mixed-integer programming, and SAT solvers.Source: §2.1 · p. 4

2.2 Linear programming gains were well-defined, steady, and faster than Moore’s Law for 21 years

• A 2x speedup every 13 months was observed in linear programming over 21 years, while Moore’s Law yielded an approximately 1500x efficiency gain.The trend’s smoothness is partly attributed to aggregating problems of varying difficulty; distinct CPLEX and Gurobi releases enabled comparisons with hardware held fixed.Source: §2.2 · p. 4, Figure 1 · p. 5

Caveats

• The solver benchmark’s 30,000-second cap makes the estimate conservative but sensitive to the cap, and its designer had an incentive to show large progress.The cap likely affected earlier, weaker solvers more often, making them appear relatively stronger.Source: Caveats · p. 4

2.3 184x reduction in training cost (in dollars) to get to ResNet-50 performance since 2017

• DawnBench submissions reduced the original $2,323 ResNet-50 training-cost benchmark by 184x, to $12.60 in September 2017.Dollar cost aggregates algorithmic gains, hardware improvements, cloud-cost reductions, and hardware utilization; targeted FLOPs measures help distinguish these effects.Source: §2.3 · p. 5

2.4 We can estimate costly-to-observe algorithmic efficiency improvements through scaling laws

• Scaling laws can estimate efficiency gains that would require prohibitively expensive compute to observe directly.The paper also notes rising AI investment, including private investment from $7B in 2012 to $40B in 2018 and public offerings and mergers/acquisitions from $5B to $23B.Source: §2.4 · pp. 5–6

3.1 Main result primarily based on existing open source re-implementations of popular models

• Most architectures used PyTorch example models and suggested hyperparameters, supplemented by existing MobileNet and ShuffleNet implementations.The authors indicate deviations from suggested hyperparameters in the following section.Source: §3.1 · p. 6
• Training compute is calculated as FLOPs per image multiplied by images per epoch and the epochs required to match or exceed the trained AlexNet model.Per-image FLOPs were counted with a PyTorch library and checked against other methods for several models.Source: §3.1 · p. 6

3.2 We made few hyperparameter adjustments between architectures and did minimal tuning

• Training generally used SGD with batch size 256, momentum 0.9, weight decay 1e-4, and 90 epochs, with suggested starting learning rates and decay schedules.A ResNet18 scan found the suggested hyperparameters performed best; AlexNet used its original 10x decay every 30 epochs, while other models used a 1000x total reduction.Source: §3.2 · pp. 6–7
• Smoothing learning-rate schedules improved early learning; without it, many models reached AlexNet performance at exactly 31 epochs, and smoothing often raised measured efficiency by 2–4x without meaningful final-performance differences.The figure shows ResNet-50 learning curves; Appendix C compares final accuracy with original-paper results.Source: §3.2 · p. 7, Figure 2 · p. 7

4.1 Key Result: 44x less compute needed to get to AlexNet-level performance

• 44x less compute was required to train to AlexNet-level performance from 2012 to 2019, an efficiency doubling time of 16 months.Figure 3 marks the lowest-compute point at each time in blue and all measured points in gray.Source: §4.1 · p. 7, Figure 3 · p. 8
• Algorithmic efficiency improved faster than Moore’s Law’s 11x cost improvement over the period, and both trends made AlexNet-level training cheaper.Source: §4.1 · p. 7
• Training-efficiency gains came from both fewer epochs and fewer FLOPs per epoch, with their contributions varying across models.The analysis is sensitive to how far the original work trained toward convergence: stopping AlexNet at 62 rather than 90 epochs would change the estimated gain from 44x to 30x.Source: §4.1 · p. 8, Table 1 · p. 9

4.2 FLOPs based learning curves can help clarify comparisons between models

• FLOPs-based learning curves show ResNet-50 dominating VGG-11 and GoogLeNet dominating AlexNet at every training-compute level.VGG-11 reached higher final accuracy than AlexNet, but required more compute to reach every performance level.Source: §4.2 · p. 9, Figure 4 · p. 9
• Translation showed more efficiency progress than vision over a shorter horizon, while the observed Go and Dota rates would surpass vision with modest 3x and 5x gains over five years.Go and Dota cover shorter time spans and more often reflect one research group, so their rates represent a different regime from image recognition and translation.Source: §4.2 · pp. 9–10
• Training and inference efficiency may improve at somewhat similar rates, although inference evidence spans fewer points across time and domains.ShuffleNet reached AlexNet-level performance with an 18x inference-efficiency increase in five years; EfficientNet-b0 reached ResNet-50-level performance with a 10x increase in three and a half years.Source: §4.2 · p. 10

5 Discussion

• The authors attribute the 44x efficiency gains to sparsity, batch normalization, residual connections, architecture search, and appropriate scaling.They report a 14x reduction from batch normalization, at least 4.3x attributed to sparsity under an assumption, and 2.1x from resizing ResNet-50 for AlexNet-level performance.Source: §5 · pp. 10–11

5.2 It’s unclear the degree to which the observed efficiency trends generalize to other AI tasks

• The observed doubling times were similar at AlexNet and ResNet-50 performance, but generalization remains uncertain, especially to reasoning tasks.Measurements focus on tasks with substantial investment and widely recognized capability progress; the authors suggest reasoning tasks might show only small efficiency gains.Source: §5.2 · p. 11

5.3 Why new capabilities are probably a larger portion of progress than observed efficiency gains

• 7.5 million times is the estimated increase in effective training compute available to the largest AI experiments from 2012 to 2018.The estimate multiplies a 300,000x AI-and-compute trend by a 25x AlexNet efficiency trend.Source: §5.3 · p. 12
• The authors argue that new capabilities may represent much larger efficiency gains than measured for existing capabilities, citing 60x efficiency for machine translation over three years.They suggest first creating a capability may require far less compute than scaling an earlier system to match it, but such gains are not generally observed empirically.Source: §5.3 · pp. 12–13
• The effective-compute synthesis is speculative, and the paper leaves forecasting whether its macro trends will continue out of scope.The authors say the observed efficiency estimates suggest possible short- and long-term expectation updates only if progress continues at a similar rate.Source: §5.3 · p. 13, Figure 5 · p. 14

5.5 It’s possible there’s an algorithmic Moore’s Law for optimization problems of interest

• Systematic measurement could clarify whether an AI analogue of Moore’s Law exists, but it would track progress across a frontier of optimization problems rather than one task.The authors see similar rates as more likely on similar tasks, while noting uncertainty about general versus domain-specific gains and how they compound across benchmarks.Source: §5.5 · p. 14

5.6 Research provides leading indicators of the future economic impact of AI

• Measures of research progress are of interest as leading indicators of AI’s eventual economic and societal impact.The paper notes that past general-purpose technologies took decades to become widespread.Source: §5.6 · p. 15

5.7 Major limitations

• The analysis uses few tasks, may undercount new capabilities, measures final-run rather than development costs, and does not forecast trend persistence.The authors identify limited generalizability, omitted development costs, and uncertainty about extrapolation as major limitations.Source: §5.7 · p. 15

• A good model of AI progress should integrate hardware and algorithmic efficiency measures because their gains multiply and neither is negligible over meaningful horizons.Source: §6 · p. 15
• The authors report that AI models for high-interest tasks are getting cheaper to train at an exponential rate faster than Moore’s Law.Source: §6 · p. 15
• The paper argues for reporting and tracking training-efficiency state of the art over time, noting that efficiency is a meaningful slice of experiments’ learning curves.Source: §6 · p. 15

Machine Translation

• 61x less compute was estimated for Transformer to reach Seq2Seq-level performance on English-to-French WMT’14 three years later.The estimate uses 3.3e18 FLOPs for Transformer base and 4.0e19 FLOPs for Seq2Seq; Transformer base reached the level at around 20% of its run.Source: A Calculations for efficiency… · p. 18
• On WMT-14 English-to-French translation, the Transformer was estimated to require 9x less compute than GMNT to reach GMNT-level performance, one year later.The estimate compares 2.3e19 FLOPs for Transformer big with 1.4e20 FLOPs for GMNT; the base Transformer reached Seq2Seq-level performance around 68% through its run.Source: A Calculations for efficiency… · p. 18

AlphaGo Zero to AlphaZero

• 8x less compute was estimated for AlphaZero to reach AlphaGo Zero-level performance approximately one year later.The estimate includes a 4.4x decrease in total training FLOPs and 390,000 of 700,000 training steps to match AlphaGo Zero.Source: A Calculations for efficiency… · p. 18

OpenAI Five Rerun

• OpenAI Five Rerun reached the same skill level from scratch on the final environment using 5x less compute two months after the original match.Some of the original run’s additional cost was attributed to a changing environment, with balance patches approximately every two weeks during its 10-month training period.Source: A Calculations for efficiency… · p. 19

• Table 3 is scoped to FLOPs required to reach the same AlexNet-level accuracy.Source: Table 3 · p. 19
• Training FLOPs are calculated as epochs multiplied by FLOPs per image and images per epoch.Images per epoch is given as 1.28 × 10^6.Source: B Calculations for efficiency… · p. 19
• The passage also gives a teraflop/s-day conversion of 1e12 multiplied by the seconds in a day.Source: B Calculations for efficiency… · p. 19

• Table 4 compares relevant models’ final top-5 training accuracy.Source: Table 4 · p. 20
• The appendix presents final training-accuracy comparisons on two distinct metrics: top-5 and top-1.Source: Table 4 · p. 20, Table 5 · p. 20
• Table 5 compares relevant models’ final top-1 training accuracy.Source: Table 5 · p. 20