PaperlayerView exact revision on arXiv

Source-linked AI summary

Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

Bruno Mlodozeniec, Spencer Frei

arXiv:2610.34514v1cs.LGstat.ML

TL;DR

Scaling-law fitting is often treated as routine, although local-search procedures do not guarantee near-optimal fits and may obscure equally fitting alternatives. The paper develops deterministic branch-and-bound procedures to certify fit quality and map near-minimizers. It concludes that near-optimal fits can imply divergent forecasts, so the fitting stage should be understood through the range of predictions rather than a single selected curve.

  • Problem

    Heuristic local optimization offers no guarantee that the returned scaling-law fit is nearly optimal and can conceal how differently near-optimal fits extrapolate.

    Source: §1 · p. 2

  • Method

    A deterministic branch-and-bound framework certifies near-optimal fitting loss and maps the near-minimizer set to prediction envelopes.

    Source: §1 · pp. 2–3

  • Results

    Near-optimal fits can yield substantially different extrapolations, so a single selected curve can obscure the range of predictions supported by the data.

    Source: §1 · p. 3, §6 · p. 18

  • Takeaways & Limitations

    The fitting stage is better characterized by the uncertainty across near-optimal curves than by an unverified single fit.

    Source: §6 · p. 18

    The analysis applies the fitting procedure to a single realized scaling ladder without accounting for observation randomness, which can change the fitted law and its sublevel set.

    Source: §4.5 · p. 16

Abstract

from arXiv · show

Estimating neural scaling laws involves an end-to-end pipeline: designing a scaling recipe, training models across scales, evaluating models on a target metric, and deploying an algorithm to fit a scaling law to the resulting data. This final fitting stage is often treated as routine: standard procedures minimize a non-convex fitting loss using local optimization from a grid of initializations, and return the best solution found. Yet such heuristic searches can fail in two distinct ways: they offer no guarantee of returning a nearly optimal solution, and they obscure whether fundamentally different parameter choices fit the data nearly as well. Indeed, we show that standard scaling law fitting is subject to a Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers correspond to broad parameter regions whose extrapolations diverge substantially at scale. We resolve these issues with a deterministic branch-and-bound algorithm that guarantees nearly optimal fitting loss and exhaustively maps the near-minimizers to the implied range of scaling curves.

1. Introduction

Scaling-law fitting can return near-optimal fits whose extrapolations diverge, making a single selected curve misleading. The paper replaces heuristic local search with certified optimization that maps near-minimizers and bounds their predictions.

1. Introduction

  • The proposed deterministic branch-and-bound algorithm uses loss bounds over parameter boxes to prune suboptimal regions and refine promising ones.A coordinate reparameterization makes uniform lower and upper bounds on log-residual fitting losses straightforward to compute.Source: §1 · p. 2
  • Near-minimizers can fit observed scaling ladders similarly yet predict widely diverging outcomes at larger scales.This Rashomon effect appears across synthetic datasets and four real empirical ladders.Source: §1 · pp. 2–3
  • For any user-specified ε > 0, the algorithm certifies a fit within ε of the global minimum over a declared compact domain.It also computes an arbitrarily tight cover of the ε-sublevel set of near-minimizers.Source: §1 · p. 3
  • At any unobserved target scale, the algorithm gives lower and upper prediction bounds across all parameter configurations in the ε-sublevel set.The bounds ensure that no near-optimal fit predicts outside the envelope.Source: §1 · p. 3

2. Background

The background formalizes fitting as choosing a power-law curve to map scale inputs to observed validation losses. It then describes the commonly used Chinchilla Approach 3 objective and its multi-start local optimizer.

2.1. The Scaling Law Estimation Problem

  • The scaling-law estimation problem maps positive scale inputs to positive validation-loss observations using a power-law hypothesis class.The paper focuses primarily on the sum-of-power-laws class associated with Hoffmann et al. (2022).Source: §2.1 · pp. 3–4
  • A fitting algorithm approximately minimizes a loss on observed scale–loss pairs, and its returned parameters are used to extrapolate to larger, unobserved scales.Common residual choices include squared error and Huber loss in log-space.Source: §2.1 · p. 4

2.2. Chinchilla Approach 3

  • Chinchilla Approach 3 minimizes log-residual Huber loss and uses multi-start local optimization to select the lowest-loss fit.For the original five-parameter setting, L-BFGS runs from a Cartesian grid of initializations; the grid boundaries and density are heuristic.Source: §2.2 · p. 4

3. Ambiguous Fits: The Rashomon Effect in Scaling Law Estimation

The section shows that scaling-law fitting can exhibit a Rashomon effect: near-optimal fits on observed data may produce substantially different extrapolations. It also explains that heuristic optimization leaves uncertainty about both fit quality and the resulting prediction envelope.

3. Ambiguous Fits: The Rashomon Effect in Scaling Law Estimation

  • The standard grid-based local search provides no lower bound on minimum fitting loss, so it cannot certify how suboptimal its returned fit is.Across four literature ladders, the envelopes generally narrowed as grid density increased, but this was not guaranteed because the grids were not nested and results depended on which basins initializations reached.Source: §3 · p. 5
  • Two fits certified within 1% of optimal loss predicted a 74.6× difference in compute needed to reach a given loss at 300× the training range.The example uses a synthetic scaling ladder and illustrates how one selected fit can obscure near-optimal alternatives.Source: §3 · p. 5
  • Even when fitting loss is virtually identical on observed data, near-minimizers can yield substantially different scaling curves.The curve returned by a heuristic optimizer is therefore not uniquely determined by the observed ladder.Source: §3 · p. 5

4. Certified Fitting and Sublevel Prediction Envelopes

This section presents certified branch-and-bound searches for near-optimal scaling-law fits, covering near-minimizing parameter regions and bounding the prediction ranges they imply. It also addresses extrapolation uncertainty from heuristic fits and computational and modeling limits on when the guarantees are practical.

4. Certified Fitting and Sublevel Prediction Envelopes

  • At the 55-initialization Chinchilla setting, equally fitting alternatives imply a 14×–146× compute-efficiency difference for a target loss.The extreme fits were found by local search, so the true prediction envelope can only be wider.Source: §4 · p. 6, Figure 2 · p. 6
  • The three algorithms certify an ε-optimal fit, cover the ε-sublevel set, and bound predictions at a test point within slack η.The cover contains S_ε/2 and is contained in S_ε; the prediction interval contains the range over S_ε and lies within the range over S_(ε+η).Source: §4 · p. 7

4.1. Branch and Bound Global Optimization

  • Branch-and-bound uses loss lower bounds to prune regions or splits them, stopping when the best candidate is within ε of the lowest remaining bound.This certifies near-global optimality because the lowest bound cannot exceed the true minimum.Source: §4.1 · pp. 7–8

4.2. The Algorithms

  • A monotone reparameterization makes box corners bound predictions and enables fitting-loss bounds that tighten as boxes shrink.The approach can extend beyond Huber loss and the specified power-law family.Source: §4.2 · pp. 9–10

4.3. Certified Sublevel Sets Algorithm

  • The certified sublevel-set algorithm returns a finite box cover containing every fit within ε/2 of optimal while guaranteeing each included parameter is within ε.Mapping the cover back to the original parameters generally yields regions that are not boxes.Source: §4.3 · p. 12
  • The certified cover reveals an elongated, curved near-optimal parameter region whose implied scaling curves diverge as scale grows, with divergence persisting at tolerances within 1% of optimal loss.Figure 5 illustrates the cover at a much larger tolerance, roughly 90 times the certified minimum loss.Source: §4.3 · pp. 13–14, Figure 5 · p. 14
  • Although Algorithms 1 and 2 share worst-case bounds, the sublevel cover is much more expensive in practice because it refines boundary boxes; it was not practical at thresholds comparable to Chinchilla suboptimality.For fixed dimension d, Phase 2 performs O(epsilon^-d) iterations.Source: §4.3 · p. 14

4.4. Certified Prediction Envelopes

  • The envelope search avoids covering all viable parameters, certifying bounds that contain every S_epsilon prediction and whose endpoints are attained in the slightly enlarged S_epsilon+eta set.Repeating the pointwise search across target scales produces an envelope containing every viable scaling curve.Source: §4.4 · pp. 14–15
  • For three-parameter laws, each envelope endpoint took under a minute to certify at 1% of optimal loss; five-parameter envelopes at tight tolerances exceeded compute budget, so Figure 2 uses local search.The five-parameter Chinchilla fits were within 10^-5 of optimal loss.Source: §4.4 · p. 16

4.5. Limitations and Future Work

  • The algorithms have worst-case costs exponential in parameter count, and certified five-parameter envelopes at tight tolerances exceed current compute; understanding scaling-ladder complexity remains open.The authors identify adaptive splitting, tighter loss bounds, and parallel implementations as possible ways to reduce runtime and memory.Source: §4.5 · p. 16
  • The analysis fits a single observed scaling ladder without modeling run-to-run randomness, which can change fitted laws and sublevel sets; the appropriate objective and noise treatment also remain open.The paper uses log-residual Huber loss, though its bounds can be extended to most other residual losses with minor algorithm changes.Source: §4.5 · p. 16
  • The broad sublevel sets imply substantially divergent predictions, but which scaling-ladder features make these landscapes broad or narrow—and when extrapolations are trustworthy—remains unresolved.Understanding this geometry could inform scaling-ladder design.Source: §4.5 · p. 16

5. Related Work

Prior work has examined variation in scaling-law estimation, fitting methods, and experimental design, while this paper focuses on whether near-equally good fits can make substantially different predictions. Related Rashomon-set methods motivate certified approximations for scaling laws, a continuous nonconvex model class.

Scaling laws and fitting

  • Scaling-law fitting practices include log–log linear regression, quadratic fits to IsoFLOP profiles followed by power-law regression, and direct nonlinear fitting of the complete loss surface.Approach 3 uses log-coordinate parameterization, Huber loss on log-residuals, and L-BFGS from multiple initializations, retaining the best observed local solution.Source: §5 · p. 17

Problems with scaling-law estimation

  • Prior analyses report poor fits to reconstructed Chinchilla observations, narrow bootstrap intervals after premature stopping, and variation associated with accounting, tuning, and parameter-counting choices.Other work finds some compute-optimal data-to-parameter conclusions fairly robust to these discrepancies.Source: §5 · p. 17

The Rashomon effect

  • The paper asks whether fits with nearly the same fitting loss can make substantially different predictions, distinct from prior work attributing variation to pipeline choices or optimization error.The Rashomon set is the set of models whose empirical loss lies within a tolerance of the best.Source: §5 · pp. 17–18

Improved fitting and experimental design

  • Basin hopping can find better fits more efficiently but remains heuristic, while VPNLS reduces nonlinear search for squared-error fitting but does not directly apply to the standard log-residual Huber objective.The paper reports an anecdotal observation that log-residual loss yields lower extrapolation error than MSE or Huber loss in untransformed residual space.Source: §5 · p. 17
  • Experimental-design work uses an approximate posterior over fitting basins to select runs by expected reduction in target-region prediction uncertainty relative to cost.Other work studies which models and checkpoints make extrapolation more reliable.Source: §5 · p. 18

The Rashomon effect

  • Prior Rashomon-set work studies prediction variation and uses exact or approximate methods for other model classes; this work certifies approximations for a continuous, nonconvex scaling-law class.Its viable-set cover algorithm is inspired by Set Inversion Via Interval Analysis, which constructs guaranteed inner and outer approximations of parameter sets.Source: §5 · p. 18

6. Conclusion

The conclusion argues that heuristic fitting can make scaling forecasts unreliable because it gives neither a near-optimality guarantee nor a unique justified curve. The framework certifies fit quality and bounds the predictions of near-optimal fits, while the resulting envelope width depends on the earlier pipeline stages that generated the data.

6. Conclusion

  • Scaling-law fitting is a consequential but largely heuristic stage, and its ambiguities can substantially undermine forecast reliability.Standard local optimization provides no near-optimality guarantee, while broad near-minimizer sublevel sets make a single returned fit an arbitrary selection.Source: §6 · p. 18
  • The framework certifies how close a fit is to optimal and bounds both the near-optimal parameter set and its complete prediction envelope instead of selecting one favored curve.The conclusion frames this as quantifying uncertainty introduced by the fitting stage.Source: §6 · p. 18
  • For a given dataset, envelope width is governed by the scaling-ladder design, training procedures, and evaluation metrics that generated the data.The conclusion says improving forecast reliability requires optimizing each pipeline stage.Source: §6 · p. 18

A. Proofs for Certified Fitting Algorithms

The proofs establish regularity and valid loss enclosures on a compact reparameterized domain, then use shrinking parameter boxes to support the certified algorithms. They also show that the enclosures become tight as box widths vanish.

A. Proofs for Certified Fitting Algorithms

  • On the compact reparameterized domain, predictions are Lipschitz continuous in the parameters, and the same conclusion holds for log-predictions.The proof obtains bounded derivatives using the compact domain and positive lower bounds on parameters.Source: A. Proofs for Certified Fitting… · pp. 21–22
  • For each parameter box, the proof bounds the fitting loss by enclosing log-predictions and applying the Huber loss to the residual intervals.The resulting lower and upper loss bounds are consistent as box side lengths tend to zero.Source: A. Proofs for Certified Fitting… · p. 22
  • The proof derives loss-enclosure widths from log-prediction widths and the global Lipschitz property of the Huber loss.This connects the prediction bounds to the fitting-loss bounds used by the algorithms.Source: A. Proofs for Certified Fitting… · p. 23
  • BranchRegion's widest-side splitting rule ensures that every box shrinks to zero width in every coordinate under indefinite subdivision.The proof bounds side lengths after successive splits and applies the rule to normalized widths.Source: A. Proofs for Certified Fitting… · p. 23

A.1. Proof of certified fitting

The certified-fitting proof maintains global upper and lower bounds on the optimum and terminates once their gap is within the requested tolerance. The resulting candidate is near-optimal, and the proof establishes finite termination with a worst-case iteration bound.

A.1. Proof of certified fitting

  • The algorithm maintains a global optimal-loss interval from the smallest active-box lower bound and the incumbent loss, and stops when its width is at most ε.This invariant supplies the certificate at every iteration.Source: §A.1 · p. 24
  • At termination, the returned candidate's loss is within ε of the global optimum, so its original parameters belong to the ε-sublevel set.The proof uses active and discarded boxes to show no point can improve on the incumbent beyond the certified tolerance.Source: §A.1 · p. 24
  • The fitting algorithm terminates after finitely many branching iterations, with a conservative worst-case bound of O(ε^-d) for fixed dimension d.The bound assumes exploration of the complete subdivision tree and does not account for pruning.Source: §A.1 · p. 25
  • The viable-set procedure also terminates after finitely many iterations, with an O(ε^-d) bound for fixed dimension, and its accepted and discarded boxes cover the domain at termination.The proof bounds subdivision-tree depth and uses the maintained coverage invariants.Source: §A.1 · p. 26
  • For prediction-envelope computation, the proof establishes finite termination and guarantees the returned endpoint within the requested η tolerance.The argument applies symmetrically to the upper and lower endpoints.Source: §A.1 · pp. 26–27

A.2. Prediction bounds at arbitrary target scales

For arbitrary target scales, the prediction-bound algorithm chooses the parameter-box corner that maximizes the prediction, accounting for how each coordinate affects it.

A.2. Prediction bounds at arbitrary target scales

  • Prediction increases with E and C_i; its direction with respect to r_i depends on the sign of s_i(x*).The maximizing corner therefore selects upper endpoints for E and C_i, and selects the r_i endpoint according to that sign.Source: §A.2 · p. 27
  • The general algorithm replaces each box’s upper prediction corner with MaxCorner(B, x*), leaving the fitting-loss bounds unchanged.MaxCorner selects the corner that maximizes the prediction at the target scale in the general case.Source: §A.2 · p. 27

B.1. Literature Scaling Ladder Datasets

The experimental suite uses four literature scaling ladders, including digitized Chinchilla and Llama 3 IsoFLOP data, DCLM pretraining and downstream evaluations, and Delphi runs with large-scale held-out evaluations.

B.1. Literature Scaling Ladder Datasets

  • The four literature datasets are Chinchilla MassiveText, Meta Llama 3 IsoFLOP, Apple DCLM Downstream Metrics, and Delphi Open Scaling Suite.Chinchilla and Llama 3 data come from the standardized Open Athena IsoFLOP collection of digitized experiments.Source: §B.1 · pp. 27–28
  • Apple DCLM includes 149 runs evaluated on the native training mixture, spanning models from 17M to 17B parameters and approximately five token-to-parameter ratios.All runs are used for two-dimensional fitting from parameters and data to loss; the true ratios vary slightly because model depth must be an integer.Source: §B.1 · p. 28
  • Delphi’s 88 runs include 81 IsoFLOP sweep runs for fitting and seven large-scale validation runs held out to evaluate extrapolations up to 300× training compute.The held-out runs reach 1.0 × 10^23 FLOPs; fitting uses only sweep runs at or below 3.64 × 10^20 FLOPs.Source: §B.1 · p. 28
  • Chinchilla MassiveText contains 123 Transformer training runs across nine IsoFLOP compute tiers, with values digitized from the original study’s figures.The extracted values were audited by Epoch AI and later standardized in the Open Athena benchmark suite (Besiroglu et al., 2024; Czech et al., 2026; Hoffmann et al., 2022).Source: §B.1 · pp. 28–29
  • Meta Llama 3 IsoFLOP comprises 133 runs across ten compute tiers, with loss points digitized from the Llama 3 Technical Report’s vector graphics.The digitization was performed by the Open Athena Foundation (Czech et al., 2026).Source: §B.1 · pp. 28–29

C. Experimental Details

The experimental details illustrate certified fitting and prediction bounds alongside local-search comparisons. On synthetic data, fits with nearly identical losses still yield sharply different extrapolations.

C. Experimental Details

  • Unless otherwise noted, figures use the four empirical ladders, the Section 2.2 fitting loss and parametric family, and certified C++ branch-and-bound with monotonic reparameterization.Source: C. Experimental Details · p. 29
  • Two L-BFGS fits differ by less than 0.12% in fitting loss yet imply a 74.6× compute-efficiency gap at 3 × 10^23 FLOPs, 300× beyond training.The better fit has loss 5.9208 × 10^-5 and is certified within 0.02% of optimum; the synthetic ladder has ten observations with 2% multiplicative log-normal noise.Source: C. Experimental Details · p. 29
  • The certified prediction band contains predictions from every fit within 1% of optimum at the evaluated compute values.At 3 × 10^23 FLOPs the band is [0.1202, 0.2392]; each edge is within 0.05% of an attainable prediction, and the band is interpolated log-log between 49 compute values.Source: C. Experimental Details · p. 29
  • The local-search comparison builds prediction envelopes from extreme fits meeting each grid density’s best-fit loss threshold.It uses multi-start constrained optimization, so the resulting band is an inner approximation of the full sublevel-set envelope.Source: C. Experimental Details · p. 30
  • In the illustrated branch-and-bound step, a box whose loss lower bound exceeds the incumbent is pruned, while the box containing the certified optimum is retained.The selected box is bisected along the exponent dimension.Source: C. Experimental Details · p. 30
  • The synthetic sublevel-set cover returns 2,695 certified parameter boxes and maps their viable set into a prediction envelope.For ten noisy observations, the certified optimum is about 2.29 × 10^-7; the cover uses inner and outer loss tolerances 10^-5 and 2 × 10^-5.Source: C. Experimental Details · p. 31

Related papers

Loading 2610.34514v1…