Source-linked AI summary
Revisiting Neural Scaling Laws in Language and Vision
Ibrahim Alabdulmohsin, Behnam Neyshabur, Xiaohua Zhai
TL;DR
The paper addresses the gap between fitting finite learning curves and accurately predicting performance at larger scale. It proposes extrapolation-based validation and the M4 estimator, which performs better than prior methods across several domains and supports a 90-task benchmark release.
Problem
Best-fitting scaling-law parameters do not necessarily extrapolate accurately, so interpolation-only validation can produce invalid parameter estimates.
Method
The paper proposes extrapolation-loss validation and a recipe for estimating scaling laws from learning curves, including the M4 estimator.
Results
M4 extrapolates better than prior estimators across image classification, neural machine translation, language modeling, and other evaluation tasks.
Takeaways & Limitations
Scaling-law parameters should be assessed by their extrapolation performance rather than fit to observed learning curves alone.
Takeaways & Limitations
The evaluation fixes ε0 to the loss at the random-guessing level, although similar results are observed when ε0 is optimized.
Abstract
from arXiv · showhide
The remarkable progress in deep learning in recent years is largely driven by improvements in scale, where bigger models are trained on larger datasets for longer schedules. To predict the benefit of scale empirically, we argue for a more rigorous methodology based on the extrapolation loss, instead of reporting the best-fitting (interpolating) parameters. We then present a recipe for estimating scaling law parameters reliably from learning curves. We demonstrate that it extrapolates more accurately than previous methods in a wide range of architecture families across several domains, including image classification, neural machine translation (NMT) and language modeling, in addition to tasks from the BIG-Bench evaluation benchmark. Finally, we release a benchmark dataset comprising of 90 evaluation tasks to facilitate research in this domain.
1 Introduction
Scaling has made empirical learning-curve extrapolation useful across deep learning, but fitting observed curves does not ensure accurate predictions beyond them. The paper therefore advocates extrapolation-based validation, introduces M4, and releases a 90-task benchmark.
- Scaling data, model size, and training schedules has improved performance across vision and NLP while enabling more efficient compute use.Learning-curve prediction also supports sample-size planning, early experiment termination, hyper-parameter search, and neural architecture search.
- Learning-curve extrapolation estimates performance at larger scale from observed pairs of scale and performance.A common approach fits parameters such as β and c in a power-law model before predicting f(x) for large x.
- Best-fitting parameters can differ from parameters that extrapolate accurately, making interpolation-only validation unreliable.The paper argues that scaling exponents yielding the best fit to finite curves may differ substantially from those that predict larger-scale behavior.
- The paper proposes validating scaling laws by extrapolation rather than reporting only best-fitting interpolating parameters.This methodology is presented as a more rigorous basis for estimating scaling-law parameters.
- M4 is evaluated across image classification, neural machine translation, language modeling, and BIG-Bench tasks.The study also examines how architecture type and size affect scaling exponents.
- The released benchmark dataset contains 90 tasks for research on scaling laws.
2 Related work
Prior work models neural scaling with several power-law function classes and develops theoretical explanations in selected settings. This paper adds M4 and evaluates it as an extrapolation-oriented estimator across multiple domains.
- Power-law scaling has been studied across image classification, language modeling, neural machine translation, and speech recognition.
- Theoretical accounts connect scaling exponents to hypothesis-space structure, loss smoothness, model size, and intrinsic data-manifold dimension.Under one deep-learning setting, the exponent is argued to satisfy c = O(1/d).
- Existing estimators include a basic power law M1, a saturating model M2, and the shifted parameterization M3.M2 adds ε∞ to represent nonzero limiting risk, while M3 was recently used in neural machine translation.
- The paper introduces M4 and reports that it outperforms the earlier estimators in extrapolation capability across several domains.
3 The Scaling Law Estimator M4
The paper introduces M4, a flexible scaling-law estimator designed to handle learning curves that include saturation and transition phases before reaching an asymptotic power-law regime. M4 reduces to the standard power-law model in the asymptotic case and extrapolates reliably across regimes.
- Motivation: Learning curves can pass through saturation, transition, and final power-law regimes as training data increases.In the first regime, performance remains near random guessing; the transitional regime improves quickly without yet following a power law.
- Empirical behavior: In the logistic-regression experiment, M2 is accurate only when data lie entirely in the power-law regime, whereas M4 works across all cutoff settings.The experiment uses d = 100 and label noise δ = 0.2, fitting models only to data beyond each indicated cutoff.
- Estimator construction: M4 models performance with a sigmoid-like form that approaches ε∞ + βx^c as x increases.The construction uses γ(x) = βx^c, with β > 0 and c < 0, while adding α > 0 for deviations from asymptotic power-law behavior.
- Estimator construction: M4 contains M2 as a special case, because setting α = 0 recovers the standard power-law estimator.When the observed learning curve lies entirely in the power-law regime, using M4 becomes equivalent to using M2.
- Estimator construction: M4 differs from the power law by O(αx^2c), allowing it to model measurements outside the asymptotic regime.The fitted parameters are α ≥ 0, β > 0, c < 0, and ε∞ > 0; ε0 is fixed to the random-guessing loss in the evaluation.
- Estimation procedure: The estimators are fitted by minimizing square-log loss, which penalizes relative errors equally across scales.The procedure uses block coordinate descent, combining closed-form least-squares updates with gradient descent for remaining parameters.
4 Validating Scaling Laws using the Extrapolation Error
The paper argues that scaling-law parameters should be validated by their extrapolation performance rather than selected solely for best fit on finite learning curves. A ViT experiment shows that nearly indistinguishable interpolating fits can imply substantially different extrapolations.
- Motivation: Best-fitting parameters on a finite learning curve do not generally produce the best extrapolation.The paper therefore evaluates scaling-law estimators using extrapolation loss rather than interpolation loss alone.
- ViT example: For ViT/B/16 pretrained on JFT-300M, M2 exponents from about c = −0.24 to c = −0.4 fit the observed curve almost equally well.The evaluation metric is 10-shot ImageNet error rate, and the learning curve is measured on subsets of JFT-300M.
- ViT example: c = −0.24 gives the best interpolation fit, whereas c = −0.40 gives the more accurate extrapolation estimate.Extrapolation is evaluated for substantially larger numbers of seen examples, x ≫ 10^9.
- Validation criterion: Extrapolation validation uses RMSE on log-loss over x uniformly sampled from [10^9, 2 × 10^9].The logarithm penalizes relative error, assigning equal importance to large and small error scales.
- Evaluation setup: The image-classification evaluation includes six architectures, comprising two model sizes from each of BiT, ViT, and MiX families.The table lists the architecture set used for the image-classification study.
5 Experiments
The experiments evaluate scaling-law estimators across vision, NMT, language modeling, and BIG-Bench tasks using held-out extrapolation. M4 generally extrapolates best, while architecture size and task domain affect estimated scaling behavior.
- Experimental setup: The evaluation uses separate learning-curve splits for estimator training and extrapolation testing across 72 image-classification, 5 NMT, 5 language-modeling, and 10 other language-related tasks.Extrapolation error is measured with RMSE.
- Image Classification: In few-shot image classification, small upstream datasets can overfit during extended pretraining, but pre-peak examples behave as bootstrapped fresh samples.The number of bootstrapped examples is used as the scaling-law independent variable.
- Image Classification: M4 extrapolates better than M1, M2, and M3 in image classification and outperforms the other methods on more than 70% of tasks.M4 also produces larger-absolute-magnitude exponent estimates than the other methods.
- Image Classification: Within each architecture family, larger models have more favorable image-classification scaling exponents, while M4 estimates larger exponents than previous methods.Figure 7 shows these exponent differences in the slopes of the learning curves.
- Neural Machine Translation (NMT): On NMT, estimators are fit through 256M sentence pairs and evaluated by predicting log-perplexity at 512M sentence pairs using extrapolation RMSE.Table 2 reports the resulting RMSE comparisons across five NMT tasks.
- Language Modeling: In language modeling, M4 and M2 perform best, with M4 tending to perform better; their exponent estimates are near −1/3 and decrease in absolute magnitude for larger models.M4 and M2 coincide when the learning curve lies entirely in the power-law regime.
- Scalable Tasks from the BIG-Bench Evaluation Benchmark: On BIG-Bench tasks, M3 and M4 perform equally best, while M1 and M2 perform equally worse, possibly because the learning curves are noisy.The evaluation covers five language-related tasks using their preferred benchmark metrics.
6 Discussion
The paper argues that scaling-law parameters should be validated by extrapolation rather than interpolation. It presents a more accurate estimator across domains and releases a 90-task benchmark for further research.
- Discussion: Best-fitting parameters do not generally extrapolate best, motivating validation based on extrapolation loss rather than interpolation alone.The paper identifies this distinction as a methodological problem in estimating scaling-law parameters.
- Discussion: The proposed recipe extrapolates more accurately than previous methods across state-of-the-art architectures and domains including vision, NMT, language modeling, and BIG-Bench.The paper also studies how architecture type and size affect scaling exponents.
- Discussion: The released benchmark dataset contains 90 evaluation tasks to facilitate research on scaling-law estimation.
Checklist
The checklist reports planned code and benchmark-data release, detailed training documentation, and publicly available experiments for reproducibility. It also records that asset licenses were not mentioned and that some datasets are proprietary.
- The paper reports no anticipated negative societal impacts from its study of scaling laws and improved estimator.
- The authors plan to release code and a benchmark dataset to facilitate research on scaling laws.
- The experiments include publicly available BIG-Bench data, while JFT-300M cannot be released because it is proprietary.
- Training details, including data, procedures, and architectures, are described in detail.
- The checklist indicates that licenses for the assets used were not mentioned.
A.1 Ablation
The ablation attributes M4’s extrapolation improvement to its sigmoid-like flexibility together with inclusion of the functions in M2. Removing α tests whether sigmoid behavior alone is sufficient.
- M4 improves extrapolation through both sigmoid-like behavior and containment of all functions in M2.
- M2’s inclusion within M4 is insufficient by itself because M2 has lower extrapolation accuracy than M4.
- The ablation removes α from M4 to test whether sigmoid-like behavior alone explains extrapolation performance.
- Figure 10 and Table 4 evaluate the estimators using 10-shot ImageNet accuracy and extrapolation RMSE.
A.2 Big Bench Learning Curves
The BIG-Bench analysis plots learning curves for five tasks using task-specific preferred scores. These curves are noisier than those in the preceding settings.
- The BIG-Bench learning curves are noisier than in previous cases.
- Learning curves are plotted for five BIG-Bench tasks during language-model pretraining with a 262M decoder-only architecture.
- The benchmark uses multiple choice grade for unit_conversion and date_understanding, and string match for the other tasks.
A.3 Image Classification Full Figures
The supplemental figures show few-shot evaluation curves across ImageNet, Birds2010, CIFAR100, and Caltech101 at 5-, 10-, and 25-shot settings.
- ImageNet: ImageNet results are shown separately for 5-shot, 10-shot, and 25-shot evaluation.
- Birds2010: Birds2010 results are shown separately for 5-shot, 10-shot, and 25-shot evaluation.
- CIFAR100: CIFAR100 results are shown separately for 5-shot, 10-shot, and 25-shot evaluation.
- Caltech101: Caltech101 results are shown separately for 5-shot, 10-shot, and 25-shot evaluation.