PaperlayerView exact revision on arXiv

Source-linked AI summary

The Computational Limits of Deep Learning

Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, Gabriel F. Manso

arXiv:2007.05558v2cs.LGstat.ML

TL;DR

Deep learning progress depends heavily on rising computational power, raising concerns about the sustainability of current trajectories. This paper’s meta-analysis and scaling analysis show that computational demands are escalating rapidly, requiring more efficient methods or alternative machine-learning approaches.

  • Problem

    The extent to which deep-learning progress depends on increasing computational power remains unclear, despite the growing technical, economic, and environmental burden.

  • Method

    The paper meta-analyzes 1,527 papers across nine application domains and examines deep learning’s statistical and computational scaling theoretically.

  • Results

    Across applications, increases in computing power are central to performance gains, while computational burden is scaling faster than known theoretical lower bounds.

  • Takeaways & Limitations

    If current trends continue, deep learning’s computational burden will become technically and economically prohibitive, requiring dramatically more efficient or alternative methods.

  • Takeaways & Limitations

    The authors do not believe that combining currently reported efficiency approaches would yet sufficiently reduce the severe computational scaling observed.

Abstract

from arXiv · show

Deep learning's recent history has been one of achievement: from triumphing over humans in the game of Go to world-leading performance in image classification, voice recognition, translation, and other tasks. But this progress has come with a voracious appetite for computing power. This article catalogs the extent of this dependency, showing that progress across a wide variety of applications is strongly reliant on increases in computing power. Extrapolating forward this reliance reveals that progress along current lines is rapidly becoming economically, technically, and environmentally unsustainable. Thus, continued progress in these applications will require dramatically more computationally-efficient methods, which will either have to come from changes to deep learning or from moving to other machine learning methods.

1 Introduction

The article’s meta-analysis finds that deep learning progress across many applications has depended centrally on rapidly increasing computational power. It argues that this computational burden is becoming unsustainable, while theory suggests the burden may exceed known lower bounds and that efficiency gains may be possible.

  • Empirical analysis: The study analyzes 1,527 papers across nine application domains and finds that computational requirements have escalated dramatically while computing increases have driven performance improvements.The domains include image classification, object detection, question answering, named entity recognition, machine translation, speech recognition, face detection, image generation, and pose estimation.
  • Implications: If current trends continue, deep learning’s growing computational burden will become technically and economically prohibitive and increase its environmental cost.The authors say the field will need either substantially more efficient deep learning or more computationally efficient machine-learning methods.
  • Theoretical explanation: Deep learning is computationally expensive by design: the flexibility enabling diverse modeling and expert-level outperformance also makes models dramatically more computationally expensive.The paper examines statistical and computational scaling in theory to explain this tradeoff.
  • Theoretical explanation: The computational burden of deep learning models is scaling more rapidly than known theoretical lower bounds, suggesting substantial efficiency improvements may be possible.This comparison indicates that observed growth is not fully explained by currently known lower bounds.
  • Historical context: Deep learning has long been computationally constrained, although specialized hardware and greater investment recently relaxed those constraints.The paper argues that rapidly scaling computational needs may again make computation limiting.

2 Deep Learning’s Computational Requirements in Theory

Theory suggests deep learning is intrinsically reliant on computing power because overparameterization must increase with training data, producing rapidly growing computational requirements. Although larger models offer greater flexibility and potential performance, even regularized methods incur substantial computational costs.

  • Overparameterization: Overparameterization improves neural-network performance, while stochastic-gradient optimization and early stopping can avoid classical overfitting in the interpolation regime.Networks may fit training data almost exactly while retaining reasonable predictions.
  • Computational scaling: At least quadratic computational scaling follows because training cost scales with parameters times data points, while parameter counts must grow with the dataset.The quadratic rate is explicitly described as an underestimate of how quickly networks may need to grow.
  • Computational scaling: Computation = Ω(Performance4) is a lower-bound estimate obtained by combining quadratic data growth for improved performance with overparameterization overhead.The estimate uses performance defined as 1/(RMSE), and actual scaling could be worse.
  • Generality of the bound: The fourth-order lower-bound result extends beyond root mean squared error to metrics converging as 1/√n, including smooth averaged losses and cross entropy.This covers most popular neural-network training losses under the stated assumptions.
  • Flexibility-performance tradeoff: Adding parameters increases flexibility and potential performance, whereas smaller models are more efficient but may omit important parameters and make lower RMSE values unattainable.Sufficiently large neural networks are universal function approximators, establishing the theoretical appeal of model size.

3 Deep Learning’s Computational Requirements in Practice

Across deep-learning applications, improved performance has depended strongly on rapidly increasing computing power, with ImageNet requiring thousands-fold more computation to halve error. The review indicates that this scaling is economically, technically, and environmentally unsustainable without substantial efficiency improvements or fundamental rearchitecting.

  • Computational growth: Cutting-edge systems’ computing use grew approximately 10× per year from 2012 to 2019, outpacing hardware and training-efficiency improvements.This growth exceeded the ≈35× total improvement from moving to GPUs and other cited efficiency gains.
  • Review methodology: The review manually examined 1,526 papers across benchmarks spanning image classification, detection, question answering, recognition, translation, speech, generation, and pose estimation.The authors collected performance and computation-burden data across these application areas.
  • Image classification: On ImageNet, halving the remaining error requires ≈5,000× as much computation, with a 95% confidence interval of ≈1,500×–≈17,500×.The baseline estimate explains 71% of the variance in ImageNet performance, with scaling between O(Performance10.6) and O(Performance14.1).
  • Image classification: Algorithmic improvement equivalent to a 10× increase in computing power over three years adds explanatory power, but performance still depends strongly on computation.The exponential functional form also shows statistically significant reliance on computing power but has less explanatory power than the preferred polynomial form.
  • Sustainability implications: Without substantial efficiency improvements, deep learning cannot reach projected benchmarks because hardware, environmental, and monetary costs become prohibitive.The paper identifies more efficient hardware, algorithms, or fundamental rearchitecting as necessary to reduce computational intensity.

4 Comparison to other scaling studies

The analysis evaluates scaling across particular deep learning benchmarks and the field’s evolving state of the art, incorporating both computational scaling and innovation over time. It argues that scaling up is a more reliable guide to future progress than scaling down, which can make rapid performance deterioration appear deceptively favorable.

  • Analytical scope: The analysis measures how performance improves with computing power across particular deep learning benchmarks, rather than aggregating results across domains.This preserves distinctions between computational burden within a task and applying deep learning to more computationally intensive tasks.
  • Analytical scope: Within-model studies explore only the model configurations their authors implement, whereas this analysis tracks performance evolution across the field.The cited examples restrict analysis to particular network architectures or language models using the Adam optimizer.
  • Scaling direction: Scaling down can yield artificially optimistic estimates because steep performance deterioration for smaller models may be mistaken for good scaling.Scaling up measures improvements to the state of the art, while scaling down measures how performance deteriorates away from the frontier.
  • Implication: Accounting for innovation and avoiding overly optimistic scaling-down estimates, the analysis is presented as a better long-term view of performance evolution as computing power increases.The comparison includes both within-model scaling and innovation over time.
  • Comparisons: NASNet reports scaling of O(Performance5.3), but its projected 6.85% error rate at XCiT-L24 computation would exceed the actual state of the art by more than 2%.The analysis interprets faster individual-model scaling as potentially reflecting rapid deterioration for smaller models.
  • Comparisons: EfficientNet remains near the performance frontier across computation levels while scaling slightly less well than the field, with p12.3 versus p12.2.These properties suggest EfficientNet may avoid the rapid-deterioration-equals-good-scaling trap.

5 Lessening the Computational Burden

The section surveys hardware specialization, network compression, architecture search, transfer methods, alternative machine-learning approaches, and combinations of these strategies for reducing deep learning’s computational burden. It concludes that current gains remain insufficient to offset severe scaling, especially because some methods reduce deployment rather than training costs and others face diminishing returns or transfer limitations.

  • Motivation: The central challenge is increasing performance without increasing computing power, or accepting stagnation as computational requirements become constraining.The section focuses on training cost rather than inference cost.
  • Increasing computing power: Hardware accelerators: 1.5× compute-per-dollar gains and 4.9× compute-per-watt gains came from TPUs between 2017–2019 and 2017–2020, respectively.Specialized hardware includes GPUs, TPUs, FPGAs, and ASICs, but specialization may face diminishing returns.
  • Reducing computational complexity: Network Compression and Acceleration: Network compression reduces prediction-time computation through pruning, quantization, or low-rank compression while retaining network performance.These approaches operate on trained networks, so they do not mitigate potentially prohibitive training costs.
  • Finding high-performing small deep learning architectures: Neural Architecture Search and Meta Learning: Transfer learning performance drops 45% without image biases, while performance drops 11 −14% even on datasets designed to mimic the original training data.These results illustrate meta learning’s dependence on the scope and similarity of the original training data.
  • Alternative machine-learning approaches: More than 23× speedup accompanied a 5% performance drop when researchers replaced neural text-classification models with non-neural approaches.Non-neural methods performed better with limited data, whereas neural methods prevailed when data was copious.
  • Combining approaches: Combining the approaches may produce larger gains, but reported individual improvements are not yet sufficient to dramatically reduce the severe computational scaling observed.The section also notes that alternative hardware approaches have not yet disrupted existing trends and that hardware specialization may encounter diminishing returns.

6 Conclusion

The explosion in computing power used by deep learning ended the “AI winter” and enabled new computer-performance benchmarks across many tasks. However, deep learning’s growing computational burden limits further improvement and will soon constrain applications as hardware gains slow.

  • Deep learning’s increased computing power ended the “AI winter” and established new computer-performance benchmarks across a wide range of tasks.
  • Deep learning’s prodigious computing appetite limits how far performance can improve in its current form, particularly as hardware improvements slow.
  • The growing computational burden of deep learning will soon constrain a range of applications.

Supplemental Materials … 7.2 Application Area: Images

The supplemental methodology examines five image applications and systematically collects model, benchmark, network, hardware, and performance data from arXiv preprints, with manual extraction and cross-checking. Image-classification results use accuracy-based measures, including top-1 and top-5 error rates.

  • 7.1 Data collection: Data are collected from arXiv, an open-access preprint archive covering fields including computer science, statistics, and electrical engineering.
  • 7.1 Data collection: Search terms for specific tasks and benchmarks identify relevant arXiv PDF preprints for information extraction.
  • 7.1 Data collection: The extraction records application area, benchmark details, paper and model information, network characteristics, and hardware usage.Network characteristics include parameters, training epochs, floating-point operations, and multiply-adds per forward pass; hardware usage includes type, GFLOPs, and processor count.
  • 7.1 Data collection: Manual review extracts information from preprints, and results are cross-checked against paperswithcode.com data.The cross-check includes authors’ papers, code, and achieved model-performance metrics.
  • 7.1 Data collection: Training compute cannot be estimated for many papers when authors omit necessary model details such as training time, even if hardware usage is reported.Hardware performance data are mostly gathered from official hardware-designers’ platforms or publicly available databases.
  • 7.2 Application Area: Images: The study examines five deep-learning applications to images: image classification, object detection, face detection, image generation, and pose estimation.
  • 7.2.1 Image classification: Image classification identifies image content from the image itself, with datasets including ImageNet, CIFAR-10/100, MNIST, SVHN, and Fashion-MNIST.
  • 7.2.1 Image classification: Top-1 error is the fraction of test images whose correct label is not the top prediction, while top-5 error excludes labels absent from the five predictions.Average accuracy is the share of correct test-set predictions, equivalently one minus the incorrect share.

Benchmark: ImageNet

ImageNet’s ILSVRC expanded visual recognition from smaller, 20-class datasets to roughly 1.5 million images across 1,000 classes. Deep learning began dominating the benchmark in 2012, when AlexNet achieved a 16.4% top-5 error rate.

  • Benchmark: ImageNet: ILSVRC provides about 1.5M images labeled across 1,000 classes, compared with PASCAL VOC’s about 20,000 images across 20 classes.The ILSVRC2010 dataset contains 1,261,406 training images.
  • Benchmark: ImageNet: Before deep learning, the ILSVRC2010 winner used SIFT, LBP, nonlinear coding representations, and stochastic SVM, achieving a 28.2% top-5 error rate.The winning NEC-UIUC system combined multiple hand-engineered representations with a stochastic support vector machine.
  • Benchmark: ImageNet: In 2012, Toronto’s SuperVision team won ILSVRC with AlexNet, achieving a 16.4% top-5 error rate; most entrants subsequently used deep learning.AlexNet marked the beginning of deep learning systems winning ILSVRC.

7.3 Application area: Text

This section surveys deep learning applications in text, focusing on named entity recognition, machine translation, and question answering. It describes their benchmarks, evaluation measures, and task formulations.

  • 7.3 Application area: Text: The section focuses on named entity recognition, machine translation, and question answering among several text-related deep learning tasks.Other listed applications include text classification, generation, summarization, sentiment analysis, emotion recognition, and part-of-speech tagging.
  • Named entity recognition: Named entity recognition identifies and tags text entities with predefined classes and is evaluated using F1, the harmonic mean of precision and recall.The section focuses on the CoNLL2003 benchmark, covering people, locations, organizations, and miscellaneous entities.
  • Machine translation: Machine translation generates a sentence in another language and has improved substantially with deep learning, with WMT and IWSLT serving as benchmarks.BLEU measures similarity between human and machine translations using modified n-gram precision and a brevity penalty, ranging from 0 to 1.
  • Question answering: Question answering generates correct answers from unstructured documents and requires machine reading comprehension and world knowledge.Popular measures include F1 and Exact Match, while SQuAD1.1 contains 107,785 question-answer pairs on 536 articles.

7.4 Application area: Sound

Speech recognition converts audio speech into text using acoustic and language models. Its performance is commonly evaluated with Word Error Rate (WER), including on the 240-hour Switchboard Hub500 benchmark.

  • Speech recognition: Speech recognition recognizes speech within audio and converts it into corresponding text through acoustic and language models.The acoustic model handles recognition within audio, while the language model converts recognized speech into text.
  • Speech recognition: Traditional speech-recognition systems use Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs) for acoustic modeling.
  • Evaluation: Word Error Rate (WER) is a popular speech-recognition performance metric based on word-level Levenshtein distance.Dynamic string alignment addresses differences in length between recognized and reference word sequences.
  • Benchmark: The ASR SWB Hub500 benchmark contains 240 hours of English telephone conversations collected by the Linguistic Data Consortium.

8 Model Analysis

The analysis highlights substantial challenges in estimating training compute because reported data are incomplete or imprecise. It uses Hardware Burden and Network Operations as complementary metrics, with regression models supporting compute–performance analysis.

  • Data limitations: Estimating the computing power used to train a model is challenging because all required data are difficult to obtain.The analysis identifies data availability as a central problem.
  • Data limitations: Many papers report only portions of the data needed for computing-power estimates.Incomplete reporting limits the inputs available for analysis.
  • Data limitations: Training time is often reported imprecisely, and hardware computing precision is generally omitted, leaving residual uncertainty in estimates.The paper notes that these reporting gaps increase model residuals and require some assumptions about hardware use.
  • Compute metrics: Hardware Burden is the main computing-power metric, while Network Operations provides an alternative based on the neural network’s total operations.For some ImageNet models, computing power could be estimated using both metrics.
  • Regression analysis: A 10% quantile regression better approximates the best performance possible at each level of computational burden than the paper’s conditional-mean regression.Figure 7 compares the two regression approaches.

9 Regression analog example

The regression analogy shows that regularization makes highly flexible, overparameterized models substantially more data-efficient, but provides much less relief from their computational costs. Deep learning therefore combines tractable sample complexity with a dependence on large amounts of data and computation.

  • Regression analogy: Regularization makes a flexible, full-dimensional model far more data-efficient than unregularized least squares, while retaining flexibility over many potential covariates.Lasso uses all potential parameters but penalizes nonzero coefficients, producing a sparse model.
  • Regression analogy: The regularized flexible model initially trails oracle and expert models, but eventually surpasses expert models that omit contributing factors as data grows.The expert model contains nine correct and one incorrect parameter, whereas the oracle contains exactly the ten correct parameters.
  • Regression analogy: Regularization substantially improves the pace of learning from data but helps much less with the computation required to improve performance.Figure 8 compares performance as sample sizes increase with average computation required to improve performance.
  • Deep-learning analogy: Deep learning succeeds by combining overparameterization for flexibility with implicit regularization that makes sample complexity tractable.Sufficiently large neural networks are described as universal function approximators, implying maximum flexibility.
  • Deep-learning analogy: The flexibility of deep learning inherently entails dependence on large amounts of data and computation, with vastly higher computation than more efficient models.The regression analogy connects flexible models’ computational burden to deep learning’s computational requirements.
Loading 2007.05558v2…