PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortizing Scaling Law Construction Costs

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar, Sam Laing, Niccolò Ajroldi, Shiwei Liu, Joaquin Vanschoren, Aaron Klein

arXiv:2609.05016v1cs.LGcs.AI

TL;DR

Dense scaling-law construction requires costly evaluation of configurations across model, data, and hyperparameter scales, although the fit depends on the best-loss envelope. The paper casts data collection as Bayesian optimization, using progressively expanding compute windows and surrogate-fantasized grid evaluations. These mechanisms recover accurate dense-grid fits at 10–100× lower compute, with predicted losses within 1.6% of the full-data fit far beyond the observed grid.

  • Problem

    Dense grids over model, data, and hyperparameter configurations make scaling-law construction computationally expensive, although fitting requires the best-loss envelope across compute scales.

  • Method

    The paper formulates scaling-law data collection as Bayesian optimization with progressively expanding compute windows and surrogate-fantasized evaluations for unobserved grid points.

  • Results

    10–100× lower compute recovers highly accurate fits, with predicted losses within 1.6% of the full-data fit at 6–7 orders of magnitude beyond the observed grid.

  • Takeaways & Limitations

    Compute-aware acquisition and surrogate-completed grids can recover dense-grid scaling-law fits without exhaustively measuring every configuration.

  • Takeaways & Limitations

    The evaluation replays existing dense grids rather than testing live discovery of a new scaling law at previously unmeasured scales.

Abstract

from arXiv · show

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to $10\text{--}100\times$.

1 Introduction

Scaling laws reduce the need to evaluate every training configuration, but fitting them still requires careful selection of observations from a costly dense grid. The paper formulates this selection as Bayesian optimization and combines compute-aware acquisition with surrogate-based grid recovery.

  • Motivation: Dense grids over architectures, data mixtures, optimization settings, token budgets, and parameter counts make scaling-law construction prohibitively expensive.Scaling laws instead extrapolate performance and optimal configurations from power-law fits across compute budgets.
  • Motivation: A sparse subset can fit the full scaling law well or poorly depending on which observations it contains, even with the same number of points.Figure 1 contrasts two four-point subsets against a fit using all 14 observations.
  • Motivation: Because only the minimum loss over independent variables is needed at each parametric point, evaluating the full dense grid is unnecessary.The loss envelope provides the relevant observations for scaling-law fitting.
  • Contribution: The framework formalizes scaling-law construction as Bayesian optimization, progressively expands the available compute window, and fantasizes unevaluated configurations with a surrogate.These mechanisms target sparse data collection while recovering the information needed for fitting.
  • Contribution: The proposed changes recover the dense-grid fit at 10–100× less cost than exhaustive evaluation.The reported efficiency combines compute slicing, surrogate fantasization, and metrics for constrained-budget comparison.

2 Related Work

Scaling-law construction shares the performance–cost trade-off of hyperparameter optimization but differs from standard settings by targeting a best-loss envelope across multiple compute scales. Existing approaches address related optimization or extrapolation problems, while this work treats envelope collection as a broader Bayesian-optimization task.

  • Scaling-law context: Scaling laws predict large-scale runs and other pipeline settings by fitting specialized parametric forms, reducing the cost of experiments that scale prohibitively.Related predictive approaches also cover hyperparameters and other pipeline settings.
  • Bayesian optimization: Bayesian optimization combines a probabilistic surrogate with an acquisition strategy that selects the next costly query using prediction and uncertainty.This performance–cost trade-off motivates applying BO to scaling-law data collection.
  • Positioning: Prior scaling-law BO work targets optimal hyperparameters, whereas this paper addresses the more general task of constructing a best-loss envelope.The cited prior approach covers only a subset of the general case considered here.

3 Framework

The framework treats scaling-law observations as a hyperparameter-optimization problem over compute, model size, token budget, and training hyperparameters. It improves efficiency by restricting acquisition to progressively larger compute windows and fitting provisional laws on surrogate-completed grids.

  • Problem formulation: Scaling-law construction seeks observations near the optimal loss min L(C) at each compute scale so the fitted law remains predictive over large extrapolation ranges.The independent variables include model size N, token budget D, and training hyperparameters λ.
  • Efficiency sources: Compared with dense evaluation, efficiency comes from avoiding low-performing configurations and approximating the optimal loss envelope before measuring the complete grid.These are the two sources of savings identified by the framework.
  • Compute slicing: Compute slicing restricts each acquisition step to a window of observed compute scales plus the immediately next slice.The surrogate is fitted on observations in Wj, while acquisition searches over the expanded Wj+1; slice size is fixed by study design.
  • Fantasized fits: Surrogate fantasization fills unobserved grid locations with predicted outputs, allowing a provisional scaling law to be fitted on observed losses plus surrogate estimates.As observations improve the surrogate, the proxy-grid fit can approach the full-data fit at lower cost.
  • Framework integration: The framework modifies vanilla Bayesian optimization with compute slicing and fantasized fits to track scaling-law improvement during acquisition.The complete formalization is provided in the paper’s framework and appendices.

4 Empirical Analysis

The experiments replay existing dense grids to compare full-space and growing-window Bayesian optimization, with and without surrogate fantasization. Growing-window acquisition with fantasization recovers accurate scaling laws much earlier and maintains close agreement under extreme extrapolation.

  • Experimental setup: The evaluation compares Full and compute Window acquisition, each with observed-only or fantasized fits, using predefined scaling-experiment grids.The goal is to recover full-grid scaling-law fits with substantially less cumulative compute.
  • Metrics: The metrics jointly assess coefficient recovery, predicted-loss error, and optimal-loss-envelope coverage against compute.The figure reports parameter errors, held-out-loss MSE, and envelope recovery; shaded bands show 1 S.D. over 10 seeds.
  • Results: 10× speedup is achieved by Window+fantasization over Full+fantasization, and up to 100× over Full+observed-only.Window discovers envelope evaluations earlier, which accounts for the observed efficiency.
  • Extrapolation: At 1% of total budget, predicted losses remain within 1.6% of the full-data fit even 6–7 orders of magnitude beyond the observed grid.The table reports relative predicted-loss differences at different fractions of the compute needed to exhaust the grid.

5 Conclusion & Future Work

The framework casts scaling law construction as Bayesian optimization, progressively expands the compute window during acquisition, and uses surrogate fantasization to approximate dense-grid fits. It achieves accurate fits at substantially lower compute cost while identifying several directions for future work.

  • Scaling law construction is formulated as a Bayesian optimization problem with progressively expanding compute windows during acquisition.This connects the acquisition process to the compute-ordered structure of practical scaling law data collection.
  • Surrogate fantasization evaluates scaling law fits on a simulated dense-grid proxy, avoiding dense measurement of every compute slice.Unevaluated configurations are assigned surrogate predictions so the broader grid can contribute to fitting without direct evaluation.
  • 10–100× compute savings are achieved relative to the exhaustive grid while recovered predicted losses remain within 1.6% of the full-data fit.The predictions remain within 1.6% even at 6–7 orders of magnitude beyond the observed grid.
  • A multi-metric evaluation suite captures scaling-law-fit efficiency beyond traditional minimum-loss regret.Ablations also support performance across different settings and parametric forms with standard Bayesian optimization formulations.
  • Future work includes specialized kernels and acquisition functions, direct uncertainty propagation, and live validation on novel large-scale data collection runs.The paper identifies live discovery at previously unmeasured scales as an ultimate stress-test.

A Limitations of the Current Study

The study is evaluated largely through replayed dense grids and uses standard Bayesian optimization components, while its compute-slicing choices expose scope boundaries for cost-aware acquisition. The framework’s notation and evaluation distinguish observed data, simulated grid coverage, and held-out compute scales.

  • Limitations: The results replay existing dense grids rather than conducting live data collection to discover laws at previously unmeasured scales.Metrics such as parameter differences also depend on having a reference fit; other scenarios are simulated through ablations.
  • Limitations: The method uses standard Bayesian optimization kernels and acquisition functions, with only simple ablations of alternatives.Potential extensions include scaling-law-form-aware kernels and more direct propagation of surrogate uncertainty into fitted coefficients.
  • Limitations: When costs are measured in FLOPs, multiple hyperparameters can share similar costs and may not change acquisition rankings appropriately.The compute-window width therefore remains tied to choices among new hyperparameters, longer training, and new model sizes.
  • Framework: Scaling-law studies define a grid over model sizes, token budgets, and hyperparameters, with each configuration mapped to a compute value.The full optimization space is G = N × D × Λ, and compute may be many-to-one across model-size and token-budget pairs.
  • Framework: Window-based search expands from the highest compute slice observed so far, rather than searching the entire grid at every iteration.Compute windows are nested prefixes of the ordered compute frontier, with consecutive differences forming compute slices.
  • Evaluation: Evaluation separates an observed pool below threshold τ from a held-out set at or above τ, then compares observed-only and observed-plus-fantasized fits.The latter combines measured losses with surrogate predictions for unevaluated pool points.

C.1 Datasets Used

The evaluation uses five datasets spanning different compute ranges, grid densities, and scaling-study designs. It fits both one-dimensional compute-loss and two-dimensional model-size/token scaling laws using best-loss configurations over unmodeled variables.

  • Datasets: Five datasets span different compute ranges, grid densities, and scaling-study designs, affecting which scaling laws can be recovered reliably.
  • Datasets: OELLM-English covers 51 of 54 possible (N, D) combinations, with geometrically spaced token budgets and high total collection cost.
  • Datasets: Porian-HP contains 14% of possible (N, D) combinations and varies learning rate, batch size, and Adam β2 across model and data scales.
  • Datasets: StepLaw covers 175 of 182 (η, b) combinations but only 23% of possible (N, D) combinations, making it useful for hyperparameter-search evaluation.
  • Datasets: OELLM-Multilingual covers all 36 possible (N, D) combinations while using a smaller hyperparameter sweep around each pair.
  • Scaling-law forms: The compute-only law fits the best-loss envelope at each compute level, whereas L(N, D) selects the lowest-loss configuration per (N, D) pair before fitting.

C.3 Metrics

The evaluation compares scaling-law recovery against reference fits rather than relying on minimum observed loss alone. It uses four metrics covering parameter agreement, predictive error, envelope recovery, and extrapolation.

  • Metrics: Four metrics evaluate scaling-law fit quality under constrained acquisition budgets.
  • Metrics: Parameter regret measures the unsigned discrepancy between the acquired fit parameters and the validation-fit parameters.
  • Metrics: Held-out error measures mean squared prediction error on the held-out frontier.
  • Metrics: Envelope recovery measures the fraction of reference-frontier points recovered by the mixed observed-and-fantasized grid.
  • Metrics: Relative Error Extrapolation compares percentage prediction error at larger computes between the acquired-frontier fit and the pool-frontier fit.
  • Evaluation setup: The grid is split at half the maximum compute, reserving the top compute slice as held out and never acquired.

D Ablations

The ablations test whether the framework generalizes across search methods, datasets, and parametric forms. They also examine the surrogate model’s kernel and acquisition-function choices.

  • Baseline configuration: The main experiments use a Gaussian-process surrogate with a Matérn kernel and Lower Confidence Bound acquisition for vanilla Bayesian optimization.
  • Ablation scope: The ablations investigate general search methods, other datasets, alternative parametric forms, and Gaussian-process modeling choices.

D.1 Improving any BO based search method

The proposed compute-window and fantasization modifications improve scaling-law search across surrogate and acquisition choices, while kernel and acquisition-function ablations identify configuration-dependent performance differences.

  • Search methods: Surrogate-based methods recover the compute-optimal envelope faster than random search, while random search benefits from compute-window acquisition but misses most optimal points.The comparison applies compute slicing and, where applicable, fantasization across Random Search, TabPFN, and xgboost-based BO.
  • Fantasization: Observed-plus-fantasized grid points augment acquired observations when fitting the scaling law.The solid-line variant uses acquired and fantasized points, whereas the dashed variant uses acquired points only.
  • Acquisition functions: LCB performs slightly better than EI and PI on the OELLM-English dataset.The study therefore presents main results using the GP+LCB combination.
  • Parametric recovery: The L(N, D) experiment extends the L(C) conclusions: a growing compute window accelerates recovery of α, β, and E and lowers held-out error at smaller cumulative compute.Fantasization is especially beneficial for early recovery of the per-(N, D) loss envelope.

D.5.1 Dataset experiments

Across benchmark datasets, the evaluation compares Full and Window acquisition with observed-only and observed-plus-fantasized fits to assess recovery efficiency as cumulative compute grows.

  • Dataset experiments: The benchmark comparison reports parameter recovery, held-out prediction error, and per-(N, D) envelope recovery for Full and Window acquisition variants.Each acquisition variant is evaluated with and without fantasization.
Loading 2609.05016v1…