Deep Learning Scaling is Predictable, Empirically

Joel Hestness; Sharan Narang; Newsha Ardalani; Gregory Diamos; Heewoo Jun; Hassan Kianinejad; Md. Mostofa Ali Patwary; Yang Yang; Yanqi Zhou

Source-linked AI summary

Deep Learning Scaling is Predictable, Empirically

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, Yanqi Zhou

arXiv:1712.00409v1Dec 1, 2017cs.LGstat.ML

TL;DR

Deep learning research needs better evidence about how training-set and computational scale affect accuracy and model size. The paper empirically measures these relationships across four domains using state-of-the-art models and expanding data shards. It finds power-law learning curves broadly across tested settings, with model improvements shifting error rather than changing exponents, while model size grows sublinearly with data.

Problem
Architecture search is unpredictable, while the relationships among training-set scale, computational scale, and accuracy require better analysis for estimating progress costs.
Method
The paper measures scaling by training hyperparameter-reduced state-of-the-art models on successively larger shards across four application domains.
Results
Power-law learning curves occur across tested domains and model settings; architecture and optimizer improvements shift intercepts without changing exponents, while model size grows sublinearly with data.
Takeaways & Limitations
Scaling curves can support model debugging, accuracy targets, dataset-growth decisions, system design, and continued computational scaling.
Takeaways & Limitations
Real applications had not yet reached the irreducible-error region, although its existence was tested on toy problems.

Abstract

from arXiv · show

Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.

1 INTRODUCTION

Deep learning progress combines architecture search, larger training sets, and greater computation, but architecture advances are comparatively unpredictable. This paper studies how data and compute scale relate to accuracy, finding broadly predictable power-law learning curves and practical implications for research and systems.

Architecture search can yield important advances, but often depends on creative reframing, unreliable epiphany, and serendipitous hyperparameter search.
Predicting generalization-error scaling could estimate the data and compute costs of advancing state-of-the-art performance.
The paper introduces a methodology for estimating generalization-error and model-size scaling across six models and four application domains.
Power-law learning curves appear across tested domains and many models, optimizers, regularizers, and loss functions.
Improved architectures and optimizers shift the power-law intercept but do not change learning-curve steepness within a domain.
Predictable accuracy and model-size scaling can support model debugging, accuracy targeting, dataset-growth decisions, and system design.

2 RELATED WORK

Prior theory and empirical studies provide general guidance on sample complexity, model capacity, and learning curves, but do not explain the paper’s real-world scaling results. The paper addresses an unresolved question about whether power-law error scaling generalizes across deep-learning domains.

Prior work analyzes generalization through theoretical bounds, expected-error estimates, and empirical scaling studies.
The paper claims to be the first broad empirical characterization of learning-curve and model-size scaling across application domains and models.
Many theoretical studies predict power-law generalization error with exponents such as −0.5 or −1 under specified assumptions.
Real-world problems in this paper exhibit unexplained exponents between βg = −0.07 and −0.35.
The community had not definitively established that power-law error scaling exists across most deep-learning domains.
Model-capacity research suggests parameter growth for fitting data, but capacity may not adequately explain generalization to new examples.
Prior empirical model-scaling evidence was sparse, including a reported βp ≈ 0.72 estimate for two language-modeling approaches.

3 MEASURING MODEL ACCURACY AND SIZE SCALING WITH TRAINING DATA SIZE

The methodology measures how state-of-the-art models behave as randomly shuffled training data is expanded across shards spanning multiple orders of magnitude. It controls model variants and searches for best-fit configurations while using a common validation set.

The study trains hyperparameter-reduced versions of state-of-the-art models on successively larger training-set shards.
State-of-the-art architectures are selected from recent work, with multiple architectures chosen in some domains for scaling comparisons.
Training sets are randomly shuffled and divided into shards spanning 2–3 orders of magnitude, with roughly 2× size steps.
A single non-overlapping validation set scores models trained on every shard to approximate generalization error with low variance.
The measured dataset size counts only input data actually observed by the model during a full training run.
Model capacity is controlled by removing potentially capacity-reducing regularization and identifying the smallest model able to overfit each shard.
Candidate models are generated by reducing architecture hyperparameters such as layer count and hidden-node count, then selecting the lowest validation loss.
Successive shards are evaluated through stochastic Monte Carlo grid search over model sizes, optimization parameters, and sometimes random seeds.

4 DATA SET AND MODEL SIZE SCALING RELATIONSHIPS

Across four domains, increasing training data produces predictable power-law improvements in generalization error, while best-fit model sizes generally grow sublinearly. Architecture and optimizer changes mainly shift error rather than changing learning-curve steepness, although scaling can diverge when optimization or capacity becomes limiting.

Cross-domain results: Across machine translation, language modeling, image classification, and speech recognition, generalization error improves with training data according to power-law relationships.The reported exponents span domains and metrics, with NMT near βg ≈−0.128 and language-model exponents around −0.09 to −0.06.
Scaling boundaries: Scaling can depart from the power-law trend when optimization becomes harder or models exhaust capacity, and exhaustive hyperparameter search may be needed to recover the trend.This divergence is reported as common across domains; speech models likewise plateau as data approaches effective capacity.
Language modeling: Language-model scaling is especially robust, but its small exponents imply that substantially more data is needed for substantial accuracy improvements.Word and character language models have βg ∈[−0.09, −0.06] and best-fit model-size growth near βp ≈0.7.
Cross-domain results: Different architectures and optimizers often retain the same learning-curve exponent, changing the intercept or relative error instead.LSTM and RHN word models have nearly identical curves, while Adam shifts character-LM error down by ∼5% relative without materially changing the exponent.
Model-size scaling: Best-fit model sizes grow sublinearly with training-set size, including βp = 0.69 ± 5% for 2-layer LSTMs and depth-5 RHNs and βp = 0.573 for ResNets.Character LMs also show sublinear growth, with βp = 0.78 for SGD and βp = 0.92 for Adam.
Image classification: Image-classification errors enter a power-law regime only after small data sets leave models near random guessing, with top-1 βg = −0.309 and top-5 βg = −0.488.For fewer than roughly 25 images per class, validation errors remain near random-guessing levels.

5 IMPLICATIONS OF GENERALIZATION ERROR AND MODEL SIZE SCALING

Predictable learning and model-size curves can support debugging, model exploration, data and computation planning, and hardware design. Their usefulness is bounded by small-data, compute, capacity, optimization, and irreducible-error limits.

Implications for practitioners and researchers: Predictable power-law learning curves can help diagnose data, architecture, and optimization problems because divergence may indicate deeper accuracy challenges.The methodology is suggested for debugging when empirical improvements depart from power-law-like behavior.
Implications for practitioners and researchers: Model improvements such as increased depth shift learning curves downward but might not improve their power-law exponent.The paper frames changing the exponent as requiring models to extract progressively more marginal information from each added sample.
Operational implications: Scaling decisions face three limits: insufficient training data, slow computation, and irreducible error.Predictable curves can project compute requirements for reaching a target accuracy, but the study encountered GPU-memory and runtime constraints at the largest scales.
Implications for practitioners and researchers: Model architecture exploration may be feasible on smaller data sets when models operate in the power-law region, allowing later scaling to larger sets.The paper cautions that the smaller set must be large enough to expose that region.
Hardware design implications: Improved FLOP rates can yield predictable relative accuracy improvements, while scaling exponents indicate how computationally scalable application domains are.These relationships can help prioritize domains for increased compute throughput.

6 CONCLUSION

The paper empirically validates power-law accuracy improvements from growing training sets across four machine-learning domains and broad model configurations. Architecture and optimizer changes shift curves without changing their exponent, while model size grows sublinearly with data size.

6 CONCLUSION: Across four domains, training-set growth produces power-law scaling in generalization error and required model size.The domains are machine translation, language modeling, image processing, and speech recognition.
6 CONCLUSION: Across tested domains, architectures, optimizers, and loss functions, power-law learning curves persist.Within each domain, architecture and optimizer changes shift curves but do not affect the exponent.
6 CONCLUSION: Model size scales sublinearly with training-set size.The conclusion presents this as a central empirical scaling relationship with implications for research, practice, and systems.

A DETAIL ON TESTED MACHINE LEARNING DOMAINS

The paper examines how data-generalization and model-size scaling vary across machine-learning domains, architectures, optimizers, and loss functions. It reports precise domain definitions and summarizes the breadth of tested configurations.

The study investigates data-generalization and model-size scaling across machine-learning domains and varied model configurations.The tested factors include domains, model architecture features, optimization methods, and loss functions.
Table 1 summarizes the domains, model architecture features, optimization methods, and loss functions included in the experiments.

A.1 NEURAL MACHINE TRANSLATION

The neural machine translation models learn mappings from source to target sentence sequences using shared word-piece vocabularies. Training minimizes conditional cross-entropy, while evaluation uses per-token error and bits-per-token.

Neural machine translation models learn mappings from source sentence sequences to target sentence sequences.The source and target domains are represented by vocabularies and their Kleene-star sequence spaces.
The models use a shared word-piece vocabulary between German and English, with 36,545 sub-word tokens before special tokens.Including special tokens, the source and target vocabularies contain 36,547 and 36,549 tokens, respectively.
Training minimizes cross-entropy loss for the conditional probability of the target sentence given the source sentence.
Evaluation reports per-token error rate and bits-per-token, with dataset size measured by English training tokens.

A.2.1 WORD LANGUAGE MODELS

Word language models process sequences of words to predict subsequent tokens, using continuous minibatching and normalized cross-entropy. Their effective history can become very long because sentence sequences are concatenated.

Word language models use continuous minibatching, concatenating an end-of-sentence marker with the next sentence.Training sequences are unrolled to length 80 for backpropagation.
The vocabulary contains 10,004 symbols, and the model maps word histories to outputs drawn from that vocabulary.The input space consists of sequences of previously seen words, while the output space is the vocabulary.
Continuous minibatching allows the effective word-history length to become very long.
The models use normalized cross-entropy based on the predicted probability of each token.The normalization count is either training-batch sequences or validation-set predicted words.

A.2.2 CHARACTER LANGUAGE MODELS

Character language models use bounded, non-continuous character sequences, while the broader experimental descriptions cover image classification and speech recognition models. These settings differ in sequence representation, objectives, and architectural assumptions.

A.2.2 CHARACTER LANGUAGE MODELS: Character language models unroll sequences to 150 characters and truncate longer sequences.Unlike word language models, their minibatching is non-continuous, so some sequences end at an end-of-sentence token.
A.2.2 CHARACTER LANGUAGE MODELS: Character models use a 98-symbol vocabulary and predict outputs from that character vocabulary.Their effective history length is at most 150 because minibatching is non-continuous.
A.2.2 CHARACTER LANGUAGE MODELS: Character models use normalized cross-entropy based on predicted character probabilities.The normalization count is either training-batch sequences or validation-set predicted characters.
A.3 IMAGE CLASSIFICATION: Image classification uses average validation cross-entropy, top-1 error, and top-5 error as reported metrics.The models are optimized with classification cross-entropy on each training image.
A.4 SPEECH RECOGNITION: Speech recognition models predict character sequences from audio spectrogram sequences with an encoder-decoder architecture.
A.4.1 DEEP SPEECH 2: The Deep Speech 2 decoder uses connectionist temporal classification, which marginalizes possible alignments under conditional independence across aligned output steps.An added blank label helps map encoder representations and output sequences to the same length.
A.4.2 ATTENTION MODEL: The attention speech model uses GRU encoders and an attention-based decoder that removes the conditional-independence assumption on output sequences.

B POWER-LAW LEARNING CURVE FOR COUNTING MODEL CLASSIFIER

The section analyzes a binary counting model under iid sampling and shows that its expected total loss follows a power law as training-set size increases. Under a fair-coin distribution, the derived exponent is −0.5.

Model setup: The model estimates each binary outcome’s probability by its proportion among i iid training samples.The input space is X = {0, 1}, and the initial analysis assumes a fair coin with Ptrue[0] = Ptrue[1] = 0.5.
Model setup: The loss is the L1 distance between predicted and true probabilities, averaged across outputs using the true distribution as weights.The proof sequence is reported to generalize to L2-norm and absolute KL-divergence, which empirically show the same power-law behavior.
Main result: Theorem 1 states that expected total loss for the fair-coin counting model is a power law with exponent −0.5.The proof enumerates all 2^i ordered training samples and associates each with the counting model’s predicted distribution.
Proof strategy: The proof expands expected loss over ordered samples, then exploits fair-coin symmetry and groups samples with identical outcome counts.Samples producing the same counts yield identical predicted distributions, enabling a counting-based simplification.
Proof strategy: For even i, the proof uses E[L_i] = E[L_{i+1}], reducing the bound to the even-indexed case.The remaining counting steps use binomial-coefficient symmetry to obtain the desired bound.