PaperlayerView exact revision on arXiv

Source-linked AI summary

How predictable is language model benchmark performance?

David Owen

arXiv:2401.04757v1cs.LGcs.AI

TL;DR

The paper asks whether benchmark performance, especially for specific tasks, can be predicted from model scaling well enough to inform capability forecasting. The paper estimates model loss from scaling laws, fits simple performance curves against scaled compute, and evaluates extrapolation using back-testing. Aggregated benchmarks in BIG-Bench and MMLU are much more predictable than individual tasks; individual-task forecasts are significantly more predictable than chance, with average error 17.8pp.

  • Problem

    The paper asks whether benchmark performance, especially for specific tasks, can be predicted from model scaling well enough to inform capability forecasting.

    Source: §1 · p. 1

  • Method

    The paper estimates model loss from scaling laws, fits simple performance curves against scaled compute, and evaluates extrapolation using back-testing.

    Source: §2 · p. 3, §2.2 · p. 3

  • Results

    Aggregated benchmarks in BIG-Bench and MMLU are much more predictable than individual tasks; individual-task forecasts are significantly more predictable than chance, with average error 17.8pp.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Compute scaling provides a promising basis for forecasting diverse benchmark capabilities, while specific-task forecasts remain challenging.

    Source: §5 · pp. 8, 10

    Comparisons are imperfect because models use different task subsets and prompt counts, while training and architecture effects are difficult to disentangle.

    Source: §5 · p. 9

Abstract

from arXiv · show

We investigate large language model performance across five orders of magnitude of compute scaling in eleven recent model architectures. We show that average benchmark performance, aggregating over many individual tasks and evaluations as in the commonly-used BIG-Bench dataset, is decently predictable as a function of training compute scale. Specifically, when extrapolating BIG-Bench Hard performance across one order of magnitude in compute, we observe average absolute errors of 6 percentage points (pp). By contrast, extrapolation for individual BIG-Bench tasks across an order of magnitude in compute yields higher average errors of 18pp. Nonetheless, individual task performance remains significantly more predictable than chance. Overall, our work suggests compute scaling provides a promising basis to forecast AI capabilities in diverse benchmarks, though predicting performance in specific tasks poses challenges.

1. Introduction

This paper asks whether benchmark performance can be forecast from model scaling, motivated by the value of anticipating capabilities and their broader effects. It extends prior mixed evidence with larger datasets, finding aggregate performance decently predictable while individual tasks remain harder but better than chance.

1. Introduction

  • Benchmark forecasts could help anticipate task automation, inform policy and regulation, and model the timing and distribution of AI’s economic effects.These benefits depend on downstream capabilities being decently predictable from scaling.Source: §1 · p. 1
  • The study evaluates predictability in BIG-Bench and MMLU using loss-scaling estimates, performance fits, and back-testing across a larger dataset than preceding work.It reports 6pp prediction error across an order of magnitude of compute in BIG-Bench.Source: §1 · p. 1, §1.1 · p. 3

1.1. Related work

  • Earlier work reached conflicting conclusions, linking aggregate predictability to scaling while highlighting abrupt or uncorrelated changes in individual tasks.Related studies variously attributed apparent emergence to evaluation metrics, architecture-dependent thresholds, or task-level unpredictability.Source: §1.1 · pp. 1–2
  • Aggregate BIG-Bench Hard and MMLU performance is decently predictable from model and data scaling, with 5–10pp error when doubling compute.Individual task performance is less predictable, but remains more predictable than chance or a simple baseline.Source: §1.1 · p. 3
  • The paper addresses these gaps by examining substantially more models and tasks than typical earlier studies and directly measuring extrapolation through back-testing.Prior work often emphasized goodness-of-fit or smaller datasets rather than prediction for future models.Source: §1.1 · pp. 2–3

2. Methods

The method estimates comparable model losses from scaling laws, fits simple performance curves against equivalent scaled compute, and evaluates extrapolation with back-testing. This design emphasizes interpretable predictability estimates rather than best-possible predictors.

2. Methods

  • The procedure estimates loss from scaling laws, fits performance against log-loss, and tests predictions on held-out points through back-testing.The approach uses simple models because datasets typically contain only tens of datapoints per task.Source: §2 · p. 3, §2.2 · p. 3, §2.3 · p. 4

2.1. Scaling laws, loss, and compute

  • Chinchilla scaling estimates comparable test losses from model size N and training-set size D, then expresses them as equivalent scaled compute.Predicted losses are comparable across models as estimates of common-setting performance, not necessarily to reported test losses.Source: §2.1 · p. 3, Figure 2a · p. 3

2.2. Fitting for extrapolation

  • The main sigmoid fit models an S-curve from random-guessing performance toward a normalized maximum, with an optional per-task offset.The paper also considers clipped linear and constant-offset baselines.Source: §2.2 · p. 3

2.3. Evaluating predictability

  • Figure 2 contrasts scaled compute across model and training-set sizes with back-tests whose errors generally grow as predictions extend farther ahead.Each back-test varies the number of held-out points to assess extrapolation distance.Source: Figure 2 · p. 4
  • Back-tests progressively hold out points to the right of the curve, measuring mean absolute error in percentage points across different prediction distances.Tasks require at least 10 entries for per-task back-testing, and up to 70% of points may be held out.Source: §2.3 · p. 4

3. Data

The data section defines BIG-Bench task metrics and human-performance context, then describes performance datasets spanning many architectures and compute scales, alongside MMLU data and prediction-error evaluation.

BIG-BENCH TASKS

  • BIG-Bench is a heterogeneous benchmark of challenging language-modeling tasks, with performance commonly normalized so random chance is 0% NPM and good performance is 100% NPM.Source: BIG-BENCH TASKS · p. 4
  • BIG-Bench uses varied preferred metrics, including multiple-choice grading, machine-translation measures, edit distance, and game-specific scores.Source: BIG-BENCH TASKS · p. 4
  • Using NPM for individual tasks may affect predictability, and task-specific functional forms could potentially improve it.Source: BIG-BENCH TASKS · p. 5
  • Human performance varies substantially across BIG-Bench tasks: average raters score around 45 NPM, experts around 80 NPM, and unlimited-time experts may reach 100 NPM.Source: BIG-BENCH TASKS · p. 5

BIG-BENCH PERFORMANCE DATASET

  • The per-task BIG-Bench dataset covers 11 architectures and 36 model sizes from 2M to 540B parameters, spanning 1e18 to 1e24 FLOP scaled compute.Source: BIG-BENCH PERFORMANCE DATASET · p. 5
  • Because task coverage and prompting vary, the analysis restricts overall BIG-Bench comparisons to BBH results and uses three- to five-shot prompting without chain-of-thought.Source: BIG-BENCH PERFORMANCE DATASET · p. 5
  • The separate aggregate BBH dataset contains 18 architectures and 44 model sizes from 80M to 540B parameters, with 1.5T to 3.5T training tokens.Source: BIG-BENCH PERFORMANCE DATASET · p. 5

3.2. MMLU

  • MMLU is included because GPT-4 results are available and its data had minimal training-set contamination, while Gemini models are excluded because their compute budget is uncertain and difficult to apportion across multimodal training.Source: §3.2 · p. 5

4. Results

Aggregate benchmark performance follows compute scaling relatively predictably, while individual task trajectories vary substantially and are harder to extrapolate.

4.1. Aggregate performance is decently predictable

  • Aggregate performance is more predictable than chance, with sigmoid fits reducing BBH and MMLU prediction errors by a factor of two or more.The sigmoid form generally performs best among the considered fits.Source: §4.1 · p. 6
  • 3.9pp average error across a doubling of compute in BBH rises to 12.5pp across two orders of magnitude scaling.Prediction is harder when using only data before 1e23 FLOP, where performance starts reliably increasing above chance.Source: §4.1 · p. 7
  • MMLU errors range from 10–20pp across 0.33–2 orders of magnitude, but starting above 2e23 FLOP reduces errors to 4pp per doubling and 17pp per order.Models begin reliably outperforming chance around 2e23 FLOP, making earlier predictions especially error-prone.Source: §4.1 · p. 7
  • The fits weight all observed data equally, which can underestimate later above-chance performance when earlier points remain near chance.Alternative fitting approaches may improve this behavior, although the sigmoid is sufficient for analyzing predictability.Source: §4.1 · p. 7
  • Forward extrapolation suggests BBH could exceed 80% around 6e25 FLOP and 90% at 5e26 FLOP, although future task-scaling changes could slow growth.Tasks already above chance are more predictable than tasks currently at or below random chance.Source: §4.1 · p. 7

4.2. Individual tasks vary substantially

  • Individual tasks show highly variable scaling, including sharp emergence, inverse scaling, and sudden performance changes that sigmoid fits may poorly capture.The human_organs_senses task is decently predicted, whereas english_proverbs and movie_recommendation are substantially more difficult.Source: §4.2 · p. 7
  • 17pp mean and 8.4pp median absolute error occur when extrapolating individual tasks 0.67 orders of magnitude in compute.Larger data series provide significant but imperfect predictability, while the most difficult cases can have much larger errors.Source: §4.2 · p. 7
  • Individual-task error rises from 11.2pp below 20% held-out performance to 16.9pp above 20%, with multiple-choice grading worsening more sharply at longer distances.Metric choice is not the only factor, since human_organs_senses uses multiple-choice grading yet is well predicted.Source: §4.2 · p. 7

5. Discussion and conclusions

Benchmark performance is fairly predictable with scale, especially in aggregate, while individual-task forecasts remain noisier and constrained by evaluation design and available data.

5. Discussion and conclusions

  • Aggregated benchmark performance is much more predictable than individual-task performance, supporting scale-based forecasting of overall model capabilities.The pattern appears in both BIG-Bench and MMLU and supports a scaling-focused view of AI development.Source: §5 · p. 8
  • Similar scaled compute yields similar benchmark performance across architectures, with discrepancies largely associated with training-data quantities and fitting choices.PaLM’s lower loss than similarly sized GPT-3 models is attributed to using more training data, while linear-log fits on an S-shaped curve can underestimate later points.Source: §5 · p. 8
  • 17.8pp average error across an order of magnitude reflects fat-tailed individual-task errors, although half of tasks remain below 8.6pp and performance is more predictable than chance.Prediction errors increase with task performance because early low-performing points tend to underestimate later performance.Source: §5 · p. 8, Figure 6 · p. 9
  • Multiple-choice grading has higher extrapolation error than exact string matching, while soft metrics such as correct-answer log-probability may improve forecasting but are rarely available.The comparison covers 116 multiple-choice tasks and 39 exact-string-match tasks.Source: §5 · p. 9, Figure 7 · p. 9
  • Forecasting is limited by imperfect task and prompt comparability, confounding between scale and training or architecture changes, and only one order of magnitude of training-set variation.The authors note that reducible-loss estimates may be biased across model generations and that scaling-law relationships may not persist under larger dataset changes.Source: §5 · p. 9
  • Future work includes alternative prompts and benchmarks, improved functional forms, task clustering, and modeling algorithmic progress, while benchmark design remains important because many tasks saturate quickly.The paper also highlights logistical difficulties in evaluating the full BIG-Bench suite and the need to relate benchmark tasks to capabilities that matter in practice.Source: §5 · pp. 9–10

A. Dataset details

The dataset covers models with varying model and dataset sizes and includes benchmark data for BBH, MMLU, and subsets of per-task BIG-Bench.

A. Dataset details

  • The dataset spans model sizes and dataset sizes and contains benchmark data for BBH, MMLU, and subsets of per-task BIG-Bench.Table 2 describes which benchmarks are represented for each dataset subset.Source: A. Dataset details · p. 12

B. Extrapolation fits using all points

Fits using all available BBH and MMLU points produce long-range forecasts, with MMLU nearing saturation and BBH projected to continue rising under current trends.

B. Extrapolation fits using all points

  • MMLU is already nearing saturation, whereas BBH is not projected to saturate for approximately three orders of magnitude under current trends.Figure 8 extrapolates fits using all available points to 1e28 FLOP.Source: B. Extrapolation fits using all points · p. 12

C. Fits against compute rather than reducible loss

Fitting benchmark performance against estimated training compute gives similar but slightly worse extrapolation than fitting against reducible loss expressed as scaled compute equivalent.

C. Fits against compute rather than reducible loss

  • Similar but slightly worse extrapolation results from fitting against estimated training compute rather than scaled compute equivalent.Older large models such as PaLM fall below trend because their compute budgets were allocated less effectively under later Chinchilla scaling laws.Source: C. Fits against compute rather than… · p. 12

D. Alternative fits for only frontier models

Frontier-model fits use only the best-performing model at each compute level to account for algorithmic and implementation variation. This changes BBH more than MMLU, but near-term BBH extrapolations increase only modestly.

D. Alternative fits for only frontier models

Loading 2401.04757v1…