Chinchilla Scaling: A replication attempt

1 Introduction

• 2× model size requires 2× training tokens for compute-optimal training, according to Hoffmann et al. (2022).Their study trained over 400 models, ranging from 70M to over 16B parameters, on 5B to 500B tokens.Source: §1 · p. 1
• Anil et al. (2023) replicate roughly proportional token–parameter scaling; Bi et al. (2024) find that higher-quality data favors a lower optimal token-per-parameter ratio.Bi et al. (2024) also find roughly proportional scaling, but emphasize its sensitivity to data quality.Source: §1 · p. 1
• Approach 3 illuminates dense-transformer scaling laws, supporting theoretical explanations (Michaud et al. 2024; Bordelon et al. 2024) and training–inference compute optimization (Sardana and Frankle 2023).These applications make the specific parametric estimates scientifically interesting beyond their implications for optimal scaling.Source: §1 · p. 1
• The replication’s fitted model differs substantially from Hoffmann et al.’s, whose fit inadequately describes reconstructed data and conflicts with scaling policies from other approaches.Their reported confidence intervals are implausibly tight given the dataset size. Their scaling policy also disagrees with the policy implied by the replication’s fit.Source: §1 · p. 1
• Approach 3 fits final pre-training loss as a parametric function of model size and training tokens; the replication uses a partially reconstructed dataset.The fitted function is L(N, D) = E + A N^α + B D^β, where N is the number of model parameters and D is the number of training tokens.Source: §1 · p. 1

• SVG parsing reconstructs Hoffmann et al.’s Figure 4 by extracting scatter-point coordinates and fill colors.The original plot encodes model size horizontally, training FLOP vertically, and loss through color.Source: §2 · pp. 1–2, Figure 2 · p. 2, Figure 1 · p. 2
• Axis labels or ticks map SVG coordinates to model size and training FLOP, but missing y-axis ticks require assuming vertically centered labels.This alignment assumption introduces possible coordinate-extraction error.Source: §2 · p. 2
• Approximately 0.01 accuracy limits loss estimates because the logarithmic 2.00–5.00 color scale contains only 256 hex values.The authors map color-scale pixels’ vertical positions to loss values, then assign each scatter point a loss using its fill color.Source: §2 · p. 2
• Up to 70% higher losses occur in some larger models at similar compute budgets; excluding five outliers does not materially change poor-fit conclusions.The top five points in this column have training-token-to-parameter ratios < 0.4. The reason for their high losses is unclear.Source: §2 · p. 2

3 Hoffmann et al.’s Approach 3 Replication attempt

• 4,000 bootstrap resamples of 240 reconstructed observations estimate parameter covariance after fitting Hoffmann et al.’s parametric law with Huber loss and δ = 10^-3.The main fit follows the original initialization grid. Five outliers are always excluded from parameter estimation but optionally included when bootstrapping covariance.Source: §3 · pp. 2–3
• p < 10^-51 without outliers and p < 10^-35 with outliers reject equality between Hoffmann et al.’s reported parameter vector and the refit.The χ^2 test assumes the original estimates are optimal and the reconstructed data form a generic or random subset, using bootstrap covariance. Individually, E and β differ significantly, whereas A, B, and α do not.Source: §3 · p. 3

3.1 Our estimates fit the data better compared to the reported and non-rounded values

• 90% of observations have lower losses under the refit than under Hoffmann et al.’s rounded parameters, conditional on correct data reconstruction.The refit’s residuals concentrate near zero, and 98% of its Huber losses fall below Hoffmann et al.’s median loss. A likelihood ratio test rejects equal fit quality with p < 10^-235.Source: Figure 3 · p. 3, §3.1 · p. 3
• Approximately 13% positive bias in finite-data reducible loss arises at D = 10^11 when β is rounded from approximately 0.2849 to 0.28.More precise parameters were recovered from comments in Hoffmann et al.’s TeX source. Using them explains the residual offset and closes most, but not all, of the fit gap.Source: §3 · pp. 2–3, §3.1 · pp. 3–4
• 40–50 nats in log likelihood favor the refit over even unrounded Hoffmann et al. parameters, with significant fit differences affecting extrapolated optimal scaling policies.The advantage persists with or without the five highest-loss outliers; Table 2 compares fits excluding them using a likelihood ratio test based on a probability distribution whose negative log likelihood equals Huber loss. Parameter-inequality tests remain significant: p < 10^-48 without outliers and p < 10^-35 when including them in covariance estimation.Source: Table 2 · p. 4, §3.1 · p. 4

3.2 Hoffmann et al. report implausibly narrow confidence intervals

• Approximately 50-fold wider than Hoffmann et al.’s, the refit’s 80% confidence interval for a implies nearly 600,000 runs would be needed for their precision.The refit estimates a standard error of 0.018 for a, corresponding to an interval width around 0.05. Hoffmann et al. report intervals of 0.454–0.455 for a and 0.542–0.543 for b despite likely having only 400–500 observations.Source: §3.2 · p. 5
• Averaging rather than summing Huber losses triggered early optimizer termination, worsening fitted parameters and spuriously narrowing bootstrap confidence intervals, as clarified by Borgeaud (2024).The resulting high loss scale caused early termination during both fitting and bootstrapping, leaving bootstrap parameters close to initialization. The replication attributes the inconsistent scaling prescriptions to this optimization failure rather than the scaling law’s functional form.Source: §3.1 · p. 4, §3.2 · p. 5, §3.3 · p. 5

3.3 Hoffmann et al.’s Approach 3 scaling policy is inconsistent with Chinchilla and our estimates

• Around 20 tokens per parameter is optimal under the refit, matching Chinchilla and Approaches 1 and 2, versus approximately 70 under Hoffmann et al.’s Approach 3.The refit’s confidence interval for a overlaps those from Approaches 1 and 2. Its functional form fits poorly only for five excluded outliers with extremely little training data relative to parameter count.Source: Figure 5 · p. 5, §3.3 · p. 5

4 Discussion

• Rounding reported parameters explains most of Hoffmann et al.’s poor fit to reconstructed data, but corrected parameters still fit significantly worse than the best same-form estimate.This poor fit is one of three potential issues identified with their Approach 3 estimates.Source: §4 · p. 5
• Many hundreds of thousands of observations would be needed for the reported confidence intervals, versus only ∼400 likely available.The reported intervals are therefore implausibly tight given the number of data points.Source: §4 · pp. 5–6
• Hoffmann et al.’s Approach 3 model implies a scaling policy inconsistent with their other approaches and their rule of thumb of 20 tokens per parameter.Source: §4 · p. 6
• Running the optimizer to convergence during estimation and bootstrapping yields more accurate point estimates and confidence intervals given the data.The original authors attribute all three issues to their optimizer stopping before convergence, Borgeaud (2024).Source: §4 · p. 6
• Reporting scaling-law exponents to several significant figures is insufficient to accurately establish the relationship involving such large parameter and token counts.Robust, reproducible estimates matter because Hoffmann et al.’s findings informed Google’s Gemini suite (Google, 2023), while their parameter estimates are scientifically important for machine learning theory.Source: §4 · p. 6
• 4–40 tokens per parameter are consistent with the replication’s fitted model within its plotted policy range for models trained on 1e26 FLOP or more.Tighter estimates before experiments at this scale could save a meaningful fraction of their compute.Source: §4 · p. 6

A.1 What if we don’t drop the five outliers?

• Including the five outliers leaves the main conclusions unchanged but makes parameter estimates more uncertain.These experiments had the lowest ratios of training tokens to parameters and were challenging to fit using the Chinchilla scaling law.Source: §A.1 · p. 6
• 6 × 10^-50 is the implied χ2 p-value for model equality, indicating substantial deviations in certain parameter estimates.The reported individual parameter differences concern E and β.Source: §A.1 · p. 6
• 2.0 × 10^-5 is the p-value for E, indicating a highly significant difference from its expected true value.Source: §A.1 · p. 6
• 2.4 × 10^-3 is the p-value for β, indicating a highly significant difference from its expected true value.Source: §A.1 · p. 6
• Table 3 compares the replication’s parameter estimates with those of Hoffmann et al. and reports bootstrapped standard errors in parentheses.Source: Table 3 · p. 6
• 25.6 tokens per parameter is the estimated optimum, with the estimated range consistent with the 20 tokens per parameter rule of thumb.This analysis retains all five previously excluded experiments.Source: §A.1 · p. 6

A.2 Likelihood ratio test

• A probability density whose negative log-likelihood equals Huber loss up to an additive constant converts loss minimization into likelihood maximization.The construction applies for any δ > 0 and is augmented with location µ and scale σ parameters.Source: §A.2 · p. 6
• 3 fits comprise one unconstrained fit and two fixing the Hoffmann et al. (2022) parameters at rounded or precise values while allowing σ to vary.The rounded values come from the paper’s body; the more precise values come from its TeX source.Source: §A.2 · pp. 6–7
• 5 degrees of freedom define the asymptotic χ^2 distribution used for likelihood-ratio p-values under the Hoffmann et al. parameter null, following Wilks (1938).The test evaluates the log-likelihood difference between the constrained and unconstrained fits.Source: §A.2 · pp. 6–7