PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Nikhil Sardana, Jacob Portes, Sasha Doubov, Jonathan Frankle

arXiv:2401.00448v3cs.LGcs.CL

TL;DR

Existing Chinchilla scaling laws account for training costs but omit inference costs. This paper modifies those laws for compute and real-world deployment costs, finding that inference-heavy settings favor smaller, longer-trained models while extreme token-per-parameter scaling improves quality but exposes limits in current fitting procedures.

  • Problem

    Chinchilla scaling laws account only for training computation, despite substantial inference costs that depend on model size and lifetime query volume.

  • Method

    The paper modifies Chinchilla scaling laws to optimize model size and pre-training data for total training-plus-inference compute and real-world costs, then validates the approach experimentally.

  • Results

    Model quality improves as tokens per parameter increase, including extreme ranges up to 10,000, while standard Chinchilla coefficient fitting overestimates the benefit of additional data in these regimes.

  • Takeaways & Limitations

    Inference-heavy practitioners, or those with limited deployment memory, can train smaller models for considerably longer than current Chinchilla-style recommendations suggest and still improve quality.

  • Takeaways & Limitations

    The method optimizes total FLOPs, but real-world inference costs differ from training costs because hardware utilization varies substantially, especially at small batch sizes.

Abstract

from arXiv · show

Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular Deepmind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal. Furthermore, we train 47 models of varying sizes and parameter counts to validate our formula and find that model quality continues to improve as we scale tokens per parameter to extreme ranges (up to 10,000). Finally, we ablate the procedure used to fit the Chinchilla scaling law coefficients and find that developing scaling laws only from data collected at typical token/parameter ratios overestimates the impact of additional tokens at these extreme ranges.

1. Introduction

LLM scaling laws traditionally optimize training costs while ignoring lifetime inference demand. This paper extends Chinchilla-style optimization to training and inference, arguing that substantial demand favors smaller models trained on more data.

  • Motivation: Training and inference costs both grow with model size and usage volume, making lifetime deployment demand relevant to model design.Training costs depend mainly on parameters and training data, while inference costs depend on parameters and lifetime query volume.
  • Motivation: Chinchilla scaling laws optimize training computation but do not account for inference costs.This omission matters because smaller models have lower inference costs, potentially offsetting their additional training computation.
  • Approach: The paper modifies Chinchilla scaling laws to jointly optimize parameter count and pre-training tokens for a target quality and inference demand.The analysis covers both compute and dollar costs and is designed for models optimized over their full lifetime.
  • Core finding: Expected inference demand of approximately 10^9 requests favors models substantially smaller and trained longer than Chinchilla-optimal models.The paper frames this as a training–inference trade-off: extra training compute can be recovered through lower inference cost.
  • Validation: Experiments test whether transformer quality continues improving at extreme tokens-per-parameter ratios, while an ablation examines scaling-law fitting beyond typical ratios.The study trains 47 models and evaluates whether the Chinchilla fitting procedure extrapolates reliably to long training runs.

2. Computational Optimality

The paper minimizes total training and inference FLOPs for a fixed quality using Chinchilla loss scaling and numerical optimization. As inference demand rises, the compute-optimal solution shifts toward smaller models trained on more tokens.

  • Objective: The optimization fixes pre-training loss as a quality proxy and minimizes the sum of training and inference FLOPs.Transformer costs are approximated as 6N FLOPs per training token and 2N FLOPs per inference token.
  • Scaling-law model: The method uses Chinchilla’s parametric loss function to relate model parameters and pre-training tokens to model quality.The relevant coefficients are fitted empirically and depend on the dataset and model architecture.
  • Assumptions: The analysis assumes inference demand can be estimated before training and is independent of model size among models with equivalent quality.This assumption makes lifetime demand available when selecting parameters and training data.
  • Optimization: Optimal parameter and token counts are computed numerically with Newton root-finding because the inference-adjusted problem is analytically intractable.The procedure converges for relevant loss and inference-demand inputs.
  • Demand dependence: When inference usage is far below pre-training tokens, Chinchilla models remain essentially compute-optimal; higher demand makes inference-adjusted solutions preferable.Figure 2 compares total FLOPs, parameter counts, and pre-training tokens between the two approaches.
  • Examples: A 7B-quality model facing 10^11 inference tokens is optimally replaced by a 6B model trained on 1.18× the Chinchilla-prescribed data.For a 30B-quality model with 10^13 inference tokens, the method instead recommends 13.6B parameters and 2.84× the data, reducing total FLOPs by 28%.

3. Experiments

The experiments test whether extreme token-per-parameter ratios can improve small models and evaluate those models using loss and downstream metrics across five Evaluation Gauntlet categories.

  • Experimental setup: 47 MPT models span 150M to 6B parameters and 10 to 10,000 tokens per parameter, probing transformer behavior at extreme data sizes.The dataset contains trillions of tokens from general web text and code; experiments use a single epoch without repeated data.
  • Evaluation: Models are evaluated with an open-source Evaluation Gauntlet covering World Knowledge, Commonsense Reasoning, Reading Comprehension, Language Understanding, and Symbolic Problem Solving.The Gauntlet uses task-specific in-context learning accuracy metrics from the Composer library.
  • Evaluation: Loss is evaluated alongside downstream metrics to characterize whether extreme token-per-parameter scaling improves general model quality.The experiments use the Gauntlet to assess downstream performance in addition to training loss.
  • Evaluation: The Gauntlet Average equally weights tasks after subtracting each task’s random baseline and normalizing its accuracy.This aggregation produces a normalized average across the evaluation tasks.

4. Results

Across extreme token-per-parameter ratios, loss and downstream quality continue improving without observed saturation, while models learn with similar efficiency per FLOP at ratios of at least 20.

  • Loss scaling: Loss continues decreasing through 10,000 tokens per parameter for the 150M model, with no evidence of flattening through 1,000 tokens per parameter for larger models.Further experiments are needed to determine behavior beyond the tested ranges.
  • Loss scaling: The experiments find no evidence supporting a critical model size below which a Chinchilla-optimal model’s quality cannot be matched.The authors qualify this conclusion because behavior could change beyond 10,000 tokens per parameter.
  • Downstream quality: The Gauntlet Average improves as tokens per parameter increase, with no observed saturation point where additional tokens stop improving performance.As loss decreases, smaller loss reductions correspond to larger downstream accuracy improvements.
  • Downstream quality: Loss and Gauntlet Average are tightly correlated, making loss a useful predictor of improvements in general model quality at scale.The reported relationship supports using existing scaling laws to understand downstream metric changes as parameters and tokens scale.
  • Compute efficiency: For token-per-parameter ratios ≥20, nearly parallel loss-versus-FLOPs trends indicate similar learning efficiency across standard and extreme training regimes.Training below 20 tokens per parameter is less computationally efficient, consistent with the Chinchilla results.
  • Downstream quality: Category-level averages show less consistent correlations with data and model sizes because some tasks leave smaller models near random-baseline performance.The paper therefore distinguishes category-level behavior from the aggregate Gauntlet Average.

5. Parametric Fitting

The authors test whether Chinchilla-style parametric scaling curves extrapolate to extreme token-to-parameter ratios. Including longer runs flattens the fitted curves, indicating that typical-ratio fits overestimate the benefit of additional data.

  • Parametric fitting: Fitting curves with progressively more extreme training runs makes the estimated scaling curves flatter.This suggests that fits based only on typical token ratios overestimate the impact of additional training data in the long-data-ratio regime.
  • Limitations: None of the fitted parametric curves accurately models the 150M model’s long-ratio training results.The authors suggest the loss function is insufficiently flexible to represent both smaller and larger models at extreme durations.
  • Implications: The original Chinchilla fits, built mostly from runs at ≤~100 tokens per parameter, do not reliably extend to extreme-duration training.Applied to long-data-ratio runs, they overestimate loss improvements from additional training data.
  • Limitations: The study uses 47 runs and models up to 6B parameters, versus 400 runs and models up to 16B in the Chinchilla study.The authors state that further work is needed to characterize scaling laws at extreme ratios and determine how much data is needed for accurate coefficient fitting.

6. Estimating Real-World Cost Optimality

The real-world cost analysis extends the training-plus-inference objective to account for hardware utilization and per-FLOP costs. Under representative deployment assumptions, cost-optimal models can be substantially smaller and more data-trained than Chinchilla-style models.

  • Cost model: The real-world objective incorporates separate MFUs for training, inference inputs, and inference outputs, plus training and inference costs per FLOP.This accounts for differing utilization and hardware economics across pipeline stages.
  • Assumptions: The cost model omits latency requirements and assumes MFU and cost per FLOP are independent of model size, configuration, and sequence length.It remains flexible enough to represent heterogeneous hardware utilization and costs.
  • Results: A 30B-Chinchilla-quality model with 1.5B inference requests costs 17% less when replaced by a 16B model trained on 3.35T tokens.The comparison assumes typical training and inference hardware costs and utilization.
  • Results: At 2T inference tokens, a Chinchilla-70B model uses only 1.3% more FLOPs but costs 36% more than an equal-quality cost-optimal model.The gap arises because inference output tokens have 50× lower MFU than training, which FLOP-only analysis does not capture.

7. Related Work

Prior work studies scaling laws, training–inference trade-offs, and alternative regimes, but differs in scope and inference-demand coverage. The paper situates its contribution among empirical replications, broader model studies, and newer theoretical analyses.

  • Scaling laws: Scaling-law research spans model quality, transfer performance, compute-limited regimes, and differences between dense and Mixture-of-Experts models.The cited literature includes empirical, theoretical, and phase-plane analyses.
  • Methodological critiques: Replication work reports implausibly narrow confidence intervals in the original Chinchilla study and argues that parametric fits on few points can have wide uncertainty.This raises a methodological concern for scaling-law coefficient estimation.

8. Conclusion

The paper incorporates inference costs into scaling-law analysis and validates the resulting preference for smaller, longer-trained models. Experiments find continued quality gains at extreme token-to-parameter ratios, while coefficient fitting can overestimate those gains.

  • As inference demand approaches pre-training data size, optimal configurations shift toward smaller models trained on more tokens.
  • Quality improves with increasing tokens per parameter in both loss and downstream metrics, supporting small models trained on sufficient data.
  • Inference-heavy or memory-constrained practitioners can train considerably longer than current literature suggests while still improving quality.
  • Chinchilla coefficient fitting overestimates the reduction from additional data in extremely data-heavy training runs.

Impact Statement

The paper aims to reduce the costs of producing large language models and thereby broaden access to capable LLM development.

  • Reducing the costs of training capable LLMs could make them more accessible to scientific researchers, industry, and the general population.

A. No Analytic Solution for Inference-Compute Optimality

After adding inference costs to the third Chinchilla law, the paper formulates compute minimization under a fixed-quality constraint but proves that no general analytic solution exists. The resulting equations require numerical root-finding in general.

  • The analysis assumes inference demand is independent of model size when model quality is fixed and can be estimated before training.
  • There is no general analytic solution for the compute-optimal model size and pre-training token count after accounting for inference costs.
  • The objective minimizes total compute for training and deployment while fixing model quality through the third Chinchilla loss law.
  • The optimization uses Lagrange multipliers over training tokens, model parameters, and a Lagrange multiplier.
  • Determining training tokens requires solving equations such as ax^-1.283+756.6x^-0.283+c=0, whose roots cannot generally be obtained analytically.
  • The fitted constants vary with dataset and architecture, and only special beta values permit reduction to a low-degree polynomial.

B.1. Compute-Optimal Results

The paper compares compute- and cost-optimal configurations with Chinchilla-style models and reports experimental results from models spanning wide sizes and token-to-parameter ratios. The experiments aggregate normalized Gauntlet performance by category, with larger-model runs constrained by available compute.

  • Compute-optimal analysis: The analysis compares training-plus-inference FLOPs for Chinchilla-style models against equal-quality compute-optimal configurations and reports total compute reductions.
  • Cost-optimal analysis: The cost analysis compares total training and inference costs for Chinchilla models with equivalent-quality cost-optimal models under specified GPU-utilization and pricing assumptions.
  • Experimental setup: The experiments train MPT-style models from 150M to 6B parameters at token-to-parameter ratios from 10 to 10,000.
  • Experimental setup: Training durations for larger models are limited by computational resources, and one 150M run at 10,000 tokens per parameter used batch size 960.
  • Evaluation: Figure 7 aggregates equally weighted, baseline-adjusted, normalized task accuracy within each Gauntlet category.
Loading 2401.00448v3…