PaperlayerView exact revision on arXiv

Source-linked AI summary

Broken Neural Scaling Laws

Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger

arXiv:2210.14891v17cs.LGcs.AI

TL;DR

Existing scaling-law forms are strictly monotonic and cannot model non-monotonic behavior such as double descent. The paper introduces BNSL, a smoothly connected piecewise-approximately-linear power-law form with configurable breaks, slopes, locations, and sharpness. BNSL accurately models and extrapolates scaling across a large, diverse set of architectures, tasks, metrics, and scaling quantities, including behaviors other forms cannot express.

  • Problem

    Existing scaling-law forms are strictly monotonic and cannot model non-monotonic behavior such as double descent.

    Source: §4 · p. 4

  • Method

    The paper introduces BNSL, a smoothly connected piecewise-approximately-linear power-law form with configurable breaks, slopes, locations, and sharpness.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    BNSL accurately models and extrapolates scaling across a large, diverse set of architectures, tasks, metrics, and scaling quantities, including behaviors other forms cannot express.

    Source: §1 · p. 2, §4 · p. 4

  • Takeaways & Limitations

    BNSL supports broader neural-scaling prediction while showing that sufficiently sharp, unobserved breaks impose limits on extrapolation.

    Source: §1 · p. 2, §6 · p. 11

    When sufficiently sharp breaks occur beyond the largest fitted scale, no current method can extrapolate behavior after that break.

    Source: §6 · p. 11

Abstract

from arXiv · show

We present a smoothly broken power law functional form (that we refer to as a Broken Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as amount of compute used for training (or inference), number of model parameters, training dataset size, model input size, number of training steps, or upstream performance varies) for various architectures & for each of various tasks within a large & diverse set of upstream & downstream tasks, in zero-shot, prompted, & finetuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, OOD detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, "emergent phase transitions", arithmetic, supervised learning, unsupervised/self-supervised learning, & reinforcement learning (single agent & multi-agent). When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models & extrapolates scaling behavior that other functional forms are incapable of expressing such as the nonmonotonic transitions present in the scaling behavior of phenomena such as double descent & the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws

1 INTRODUCTION

Existing scaling laws aim to extrapolate neural-network performance from smaller experiments, but many relevant metrics exhibit inflections and non-monotonic behavior that prior forms cannot express. BNSL addresses this gap and accurately models and extrapolates diverse scaling behaviors.

Motivation

  • Neural scaling laws use smaller, cheaper experiments to predict large-scale model behavior and identify architectures, algorithms, and datasets that scale best.This matters because compute, model size, and dataset resources are expected to keep increasing beyond current organizational access.Source: §1 · p. 1

Problem

  • Prior functional forms often fail on downstream and other metrics that show inflections or non-monotonic behavior, including double descent and arithmetic.Monotonic forms cannot represent temporary performance decreases followed by later improvement or delayed sharp transitions.Source: §1 · pp. 1–2

Contribution

  • BNSL accurately models and extrapolates scaling across diverse architectures, tasks, and zero-shot, prompted, and fine-tuned settings.The evaluation spans vision, language, audio, video, multimodal learning, robustness, retrieval, arithmetic, and reinforcement learning, among other areas.Source: §1 · p. 2
  • Compared with other neural-scaling forms, BNSL produces considerably more accurate extrapolations on the paper’s broad task set.It also captures non-monotonic double-descent transitions and delayed, sharp arithmetic inflections.Source: §1 · p. 2

2 THE FUNCTIONAL FORM OF BROKEN NEURAL SCALING LAWS

BNSL generalizes power laws into smoothly connected piecewise-approximately-linear segments on log-log plots. Its parameters specify the asymptote, offset, segment slopes, break locations, and break sharpness, while empirical results indicate accurate extrapolation despite the form’s broad fitting capacity.

Functional form

  • BNSL represents a scaling curve as smoothly connected approximately linear power-law segments in log-log space, with breaks between segments.Figure 1 illustrates four segments separated by three breaks, with the first two smooth and the third sharp.Source: §2 · pp. 2–3, Figure 1 · p. 2

Inputs and outputs

  • The metric y can include prediction error, cross entropy, calibration error, AUROC, BLEU, F1, reward, Elo, or FID, while x can be compute, parameters, data, input size, steps, or upstream performance.The constants are estimated by fitting the functional form to observed (x, y) data points.Source: §2 · p. 2

Parameter interpretation

  • The number of breaks n determines n + 1 approximately linear segments, while c_i changes adjacent slopes and d_i locates each break.The sharpness parameters f_i control whether transitions are narrow and sharp or broad and smooth.Source: §2 · p. 3

Extrapolation

  • Although sufficiently many segments can fit any smooth univariate function, the authors report that BNSL often needs only a small number of breaks and extrapolates neural scaling behavior accurately.This addresses the central uncertainty about whether a flexible piecewise form would generalize beyond interpolation.Source: §2 · p. 3

3 RELATED WORK

Prior work established power-law scaling across data, model size, compute, and transfer settings, while related smoothly broken forms were narrower or unevaluated. BNSL extends this family to multiple breaks and diverse scaling inputs, with its use also informed by astrophysical applications.

Prior scaling laws

  • Earlier studies used power-law forms to relate performance to dataset size, model size, compute, upstream performance, and shifted power-law inputs.Alabdulmohsin et al. (2022) also introduced a benchmark dataset used in the experiments.Source: §3 · p. 3

Related broken forms

  • Hernandez et al. (2021) described a single-break form for model-size scaling in fine-tuning and transfer, but did not fit, verify, or generalize it across other scaling inputs.They also did not identify it as a smoothly broken power law or discuss multiple breaks.Source: §3 · p. 3

Inspiration

  • Astrophysics commonly uses smoothly broken power-law forms, which inspired investigating their applicability to diverse deep-neural-network scaling phenomena.The paper presents BNSL as a broader neural-scaling application of this functional-form family.Source: §3 · p. 3

4 THEORETICAL LIMITATIONS OF PREVIOUSLY PROPOSED SCALING LAWS

Previously proposed forms have theoretical expressiveness limits: M1–M3 are monotonic and lack linear-linear inflection points, while M4 can express inflections but not non-monotonicity. BNSL is introduced as sufficiently expressive for both behaviors.

BNSL comparison

  • BNSL can produce non-monotonic behavior and inflection points, providing the extra expressiveness needed for empirical scaling phenomena.The paper points to Figures 4 and 5 as examples of these behaviors being modeled and extrapolated.Source: §4 · p. 4

M1–M3 limitations

  • M1, M2, and M3 are monotonic and cannot express inflection points because their first derivatives do not change sign and their second derivatives lack roots.These restrictions prevent them from modeling non-monotonic curves or linear-linear inflections.Source: §4 · p. 4

M4 limitations

  • M4 cannot express non-monotonic functions because its inverse mapping would require one y to correspond to multiple x values.Its deterministic mapping from y to x rules out that behavior.Source: §4 · p. 4

M4 expressiveness

  • M4 can model inflection points, unlike M1–M3, because inflections are preserved under the relevant inverse-function relationship.The paper gives a parameter setting demonstrating this capacity.Source: §4 · p. 4

5 EMPIRICAL RESULTS: FITS & EXTRAPOLATIONS OF FUNCTIONAL FORMS

Across vision, language, reinforcement learning, and non-monotonic or inflection-point settings, BNSL accurately models and extrapolates scaling behavior, often outperforming alternative functional forms.

5.1 VISION / 5.2 LANGUAGE

  • BNSL achieves the best extrapolation on 69.44% of large-scale vision tasks and 75% of language tasks, versus 13.89% and 10% for the next-best form.These evaluations use RMSLE across downstream vision, downstream language, and upstream language tasks.Source: §5.1 · p. 5, Table 2 · p. 5, Table 3 · p. 6
  • BNSL accurately extrapolates downstream performance beyond the largest fitted scale, including more than an order of magnitude and, in one setting, over 100,000 times larger.The larger extrapolation uses downstream performance with the x-axis extending beyond the fitted range.Source: §5.1 · p. 5, §5.2 · p. 7, Figure 2 · p. 8

5.3 REINFORCEMENT LEARNING

  • BNSL accurately models and extrapolates reinforcement-learning scaling across algorithms, environments, compute, dataset size, model parameters, and test-time compute.Examples include AlphaZero on Connect Four, PPG on StarPilot, PPO on Heist, and an RLHF-finetuned language model.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 NON-MONOTONIC SCALING

  • BNSL models and extrapolates non-monotonic Transformer scaling during double descent, which other functional forms are mathematically incapable of expressing.The double-descent examples use neural machine translation and include a two-break fit.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 INFLECTION POINTS

  • BNSL accurately models and extrapolates the delayed sharp inflection point in Transformer 4-digit addition as training dataset size increases.The study also reports analogous inflection-point extrapolation with training steps and model parameters on the x-axis.Source: §5.5 · p. 10, Figure 5 · p. 10

6 THE LIMIT OF THE PREDICTABILITY OF SCALING BEHAVIOR

BNSL’s extrapolation accuracy is limited when scaling behavior contains sufficiently sharp breaks that are not adequately represented in the fitted data.

6 THE LIMIT OF THE PREDICTABILITY OF SCALING BEHAVIOR

  • For 4-digit addition, accurate extrapolation requires training runs with dataset size at least 720, although the break occurs around 415.A noiseless simulation reduces the required maximum fitted dataset size to approximately 415, close to the break.Source: §6 · p. 10
  • Even infinitely many averaged training runs cannot guarantee perfect extrapolation when the fitted range ends too far before a sufficiently sharp break.The limit concerns the maximum x-axis value represented among the fitting points.Source: §6 · p. 11
  • A sufficiently sharp break beyond the fitted range currently prevents extrapolation after that break, while points below an already-covered break may add little unless specific conditions apply.The paper notes that earlier points can sometimes improve extrapolation, with examples in Appendix A.7.Source: §6 · p. 11

7 DISCUSSION

The paper presents BNSL as broadly accurate across neural-network scaling settings and argues that smoothly broken power laws may capture many such behaviors, while noting compute-related access and sampling concerns.

7 DISCUSSION

  • Across diverse architectures and upstream and downstream tasks, BNSL produces considerably more accurate extrapolations than other functional forms.The scope includes vision, language, audio, video, multimodal learning, robotics, safety-related tasks, arithmetic, and reinforcement learning.Source: §7 · p. 11
  • BNSL also expresses non-monotonic transitions and delayed sharp inflection points that other functional forms cannot express.The paper identifies double descent and arithmetic as examples.Source: §7 · p. 11

ETHICS STATEMENT

  • Accurate BNSL fitting may require many performance samples, which can be costly and may give compute-rich organizations more accurate extrapolations.The paper connects this sampling burden to a possible foresight gap between Big Tech and academia.Source: §7 · p. 11

A.1 ANALYSIS AND EXPLANATION OF WHY BNSL IS SMOOTHLY CONNECTED PIECEWISE (APPROXIMATELY) LINEAR FUNCTION ON A LOG-LOG PLOT

BNSL becomes a smoothly connected piecewise approximately linear function on a log-log plot after rewriting it in terms of z = log(x). Its summed terms resemble softplus functions, whose parameters control the curve’s shape.

A.1 ANALYSIS AND EXPLANATION OF WHY BNSL IS SMOOTHLY CONNECTED PIECEWISE (APPROXIMATELY) LINEAR FUNCTION ON A LOG-LOG PLOT

  • On a log-log plot, BNSL is approximately piecewise linear with smooth connections between segments.This follows by treating y as a function of z = log(x) and applying logarithms to Equation 1.Source: §A.1 · p. 12
  • BNSL terms resemble softplus functions, smoothly interpolating between a constant function and the identity.The parameters c_i, d_i, and f_i influence the resulting curve shape.Source: §A.1 · p. 12

A.2 DECOMPOSITION OF BROKEN NEURAL SCALING LAW INTO POWER LAW SEGMENTS THAT IT IS COMPOSED OF

A BNSL with three breaks can be decomposed into the power-law segments composing it, enabling the segments overlaid in Figure 1 and extending naturally to n breaks.

A.2 DECOMPOSITION OF BROKEN NEURAL SCALING LAW INTO POWER LAW SEGMENTS THAT IT IS COMPOSED OF

  • A three-break BNSL can be decomposed into its constituent power-law segments, a pattern that extends straightforwardly to n breaks.The decomposition supports the Figure 1 overlays when f is not too large and a is subtracted from Equation 1’s right-hand side.Source: §A.2 · p. 12

A.5 EXPERIMENTAL DETAILS OF SECTIONS 5.5 AND A.10

This section details BNSL fitting and validation procedures and evaluates extrapolation across diverse scaling axes, architectures, tasks, and evaluation settings, including audio, language, coding, vision, translation, and BIG-Bench. It also examines extrapolation beyond fitted scales, the use of pre-break information, and factors affecting accuracy such as seed variance and fitting coverage.

A.6 EXPERIMENTAL DETAILS OF FITTING BNSL AND DETERMINING THE NUMBER OF BREAKS

  • BNSL is fit by grid-search initialization followed by nonlinear least squares minimizing mean squared log error, while validation selects the number of breaks or a useful crop point.Holding out the largest fitting points provides validation data; cropping can improve extrapolation when sharp breaks make earlier points unhelpful, and small break sharpness can hinder visual identification.Source: §A.6 · p. 13

A.7 EXAMPLES IN WHICH FITTING POINTS BEFORE A BREAK IMPROVES EXTRAPOLATION EVEN THOUGH THE MAXIMUM (ALONG THE X-AXIS) OF THE POINTS USED FOR FITTING IS LARGER (ALONG THE X-AXIS) THAN THAT BREAK

  • BNSL accurately extrapolates when fitting data contain only one point beyond the largest break, integrating pre-break information where post-break-only fitting is unidentifiable.The appendix also reports extrapolation more than an order of magnitude beyond the largest fitting scale for MLP-Mixer downstream ImageNet classification.Source: §A.7 · p. 13, §A.8 · p. 14, Figure 6 · p. 14

A.9 EXPERIMENTAL RESULTS THROUGH A.14

  • BNSL yields more accurate extrapolations than M3 for compute-optimal downstream vision scaling, and accurately extrapolates scaling with training steps, diffusion images, video generation, Pareto-optimal data pruning, and upstream performance.The reported settings include ViT or MLP-Mixer pretraining, ImageNet evaluation, 4-digit addition, diffusion FID/NLL, video cross-entropy, and downstream ImageNet performance.Source: §A.9 · p. 15, Table 6 · p. 15, §A.10 · p. 15

A.15 EXPERIMENTAL RESULTS THROUGH A.24

  • BNSL accurately extrapolates downstream language and retrieval scaling across model parameters, retrieval-dataset size, input context length, sparse-model parameters, adversarial robustness, finetuning data, calibration, RLHF alignment, continual learning, and robotics.Language-parameter extrapolations can extend beyond an order of magnitude, but high seed variance may require more than seven fitting points; robotics covers both unseen- and seen-color evaluation.Source: §A.14 · p. 18, §A.16 · p. 19, §A.17 · p. 19

A.25 EXPERIMENTAL RESULTS THROUGH A.31

  • BNSL accurately extrapolates quantized models, distillation, out-of-distribution detection, molecular force prediction, math-word-problem solving, fairness and ensembles, and test-time inference compute.Distillation and out-of-distribution detection include extrapolation beyond an order of magnitude past the largest fitting scale.Source: §A.25 · p. 24, §A.26 · p. 24, §A.27 · p. 25

A.32 EXPERIMENTAL RESULTS THROUGH A.33

  • BNSL also accurately extrapolates recurrent-neural-network scaling and downstream performance from multimodal contrastive learning, extending the demonstrated scope beyond the preceding architectures and tasks.The recurrent setting evaluates ARC Challenge error, while contrastive learning evaluates zero-shot and linear-probe ImageNet performance.Source: §A.32 · pp. 27–28, Figure 30 · p. 27, Figure 31 · p. 28

A.34 EXTRAPOLATION RESULTS FOR DOWNSTREAM PERFORMANCE ON AUDIO TASKS

  • BNSL accurately extrapolates downstream zero-shot multilingual speech-recognition performance on FLEURS as model parameters increase.The data come from Whisper, pretrained on 681,070 hours of audio.Source: §A.34 · p. 28, Figure 32 · p. 28

A.35 EXTRAPOLATION RESULTS FOR DOWNSTREAM PERFORMANCE (E.G. ON LANGUAGE/CODING TASKS) TO SCALES THAT ARE GREATER THAN 100,000 TIMES LARGER THAN THE MAXIMUM (ALONG THE X-AXIS (E.G. AMOUNT OF COMPUTE USED FOR TRAINING)) OF THE POINTS USED FOR FITTING

  • BNSL accurately extrapolates downstream language/coding performance to scales greater than 100,000 times the maximum x-axis scale used for fitting.The evaluation is zero-shot test bits per word on OpenAI’s held-out internal codebase, using data from GPT-4 experiments.Source: §A.35 · p. 29, Figure 33 · p. 29

A.36 PLOTS OF BNSL EXTRAPOLATIONS ON SCALING LAWS BENCHMARK OF

  • The cited benchmark plots report BNSL extrapolation results for downstream Birds 200, CIFAR-100, Caltech101, BIG-Bench, ImageNet, and language modeling; the first five use pretraining dataset size on the x-axis.Source: Figure 34 · p. 30, Figure 35 · p. 31, Figure 36 · p. 32
  • Unsatisfactory Caltech101 and BIG-Bench extrapolations are attributed to fitting maxima near or before a break and too few points for the SciPy fitter to distinguish breaks from noise.The explanation is based on eyeballing and is discussed further in Section 6.Source: Figure 36 · p. 32, Figure 37 · p. 33
  • The benchmark set also includes language-modeling and neural-machine-translation extrapolation plots.These plots are referenced as providing further results from Section 5.2.Source: Figure 38 · p. 33, Figure 39 · p. 33
Loading 2210.14891v17…