PaperlayerView exact revision on arXiv

Source-linked AI summary

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, Jörg Franke, Sampo Pyysalo, Jenia Jitsev, Aaron Klein

arXiv:2608.28308v1cs.LGcs.AI

TL;DR

The paper addresses how learning rates, batch sizes, and loss scaling should be characterized for dense LLM pretraining across model and data scales. It studies joint and marginal hyperparameter scaling, annealing under WSD, and interacting performance-scaling forms. The experiments find that the Skaling approach better extrapolates held-out points than the Chinchilla law, while optimal hyperparameters shift after decay.

  • Problem

    The study asks how optimal learning rates and batch sizes, annealing, and loss depend on model capacity and data scale in dense LLM pretraining.

  • Method

    The paper analyzes extensive pretraining runs, modeling joint and marginal hyperparameter scaling, comparing stable and decay phases, and evaluating loss-scaling forms with model-data interaction.

  • Results

    RMSE is 0.0056 versus 0.0174 for the extended versus Chinchilla model on held-out 1.7B points, while decay favors learning rates up to ~4× larger.

  • Takeaways & Limitations

    The study establishes a baseline scaling procedure for future OpenEuroLLM models and provides an open collection of pretraining runs for reproducing the analyses.

  • Takeaways & Limitations

    Small-scale scaling laws extrapolate over roughly an order of magnitude, but their validity at substantially larger scales remains unclear.

Abstract

from arXiv · show

We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling \textit{jointly optimal} learning rates and batch sizes, we investigate their \textit{marginal} evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size \textit{transfer} between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study.

1 Introduction

The report studies how optimal learning rates and batch sizes scale, distinguishing joint from marginal behavior and examining annealing effects. It also evaluates scaling-law forms for performance across model and data scales.

  • Scaling optimal hyperparameters: The study distinguishes jointly optimal from marginal scaling of learning rate and batch size.It develops models for both behaviors.
  • Learning-rate annealing: Learning-rate annealing is analyzed under a Warmup-Stable-Decay schedule across hyperparameter settings, models, and data budgets.
  • Performance prediction: Scaling laws are used to characterize performance changes with model and data scale, including undertrained and overtrained regimes.The report evaluates functional forms that model interactions between model and data scale.
  • Report structure: The report is organized around experiments, hyperparameter scaling, annealing, cross-entropy loss, downstream evaluation, and conclusions.

2 Experimental Setting

The experiments train dense decoder-only transformers on Nemotron-CC data using AdamW and a Warmup-Stable-Decay schedule. They vary model size, token budget, learning rate, and batch size, and release the resulting pretraining runs.

  • Data: Training uses up to 300BT tokens from the high-quality Nemotron-CC subset with a shared validation set.The validation set contains 204,800 sequences, or 0.838BT.
  • Models: Models range from 47M to 1.7B parameters and use a common dense decoder-only transformer design.Architectural details include a 50,304-word vocabulary, GLU activations, QK normalization, and tied input-output embeddings.
  • Training configuration: Training uses AdamW with a Warmup-Stable-Decay schedule and intermediate decays across target token budgets up to 300BT.The learning rate decays over the final 20% of each target budget to 10^-5; the schedule uses a fixed 2000-step warmup.
  • Hyperparameter grid: The hyperparameter grid varies log2-spaced batch sizes from 2^4 to 2^10 and learning rates from 0.00025 to 0.004.Combinations with warmup exceeding 30% of the token budget are excluded.
  • Open-source resources: The complete collection of pretraining runs, checkpoints, losses, and downstream evaluations is released for reproducing the analyses.

3 Scaling Hyperparameters

The study estimates jointly optimal and conditional learning-rate and batch-size scaling laws by smoothing noisy validation-loss surfaces, then evaluates their predictive accuracy on held-out models. Jointly optimal batch size increases with data and weakly with model size, while conditional learning rate decreases with data and model size, with larger batches shifting the optimum upward.

  • 3.1 Smoothing: Loss smoothing provides a continuous, lower-variance estimate of optimal learning rate and batch size from coarse, noisy measurements.The authors fit a quadratic loss surface in learning rate and batch size using OLS for each model size and data budget.
  • 3.2 Scaling Jointly Optimal Learning Rate and Batch Size: The jointly optimal batch size explains 93.6% of observed variance, scaling strongly with data (0.46) and weakly positively with model size (0.145).The fitted relationship is evaluated using power-law forms for jointly optimal hyperparameters.
  • 3.2 Scaling Jointly Optimal Learning Rate and Batch Size: The jointly optimal learning rate explains 79.2% of variance, scaling with model size (0.417) and more weakly with data (0.0862).The data-scale effect remains statistically significant after bootstrap testing, with p-value 0.0055.
  • 3.3 Scaling Optimal Learning Rate with Arbitrary Batch Size: Conditional on batch size, the optimal learning rate decreases with model size and data, while larger batches shift the optimum upward.The data-scaling effect weakens substantially at larger batch sizes, and the conditional trend differs from the jointly optimized trend.

4 LR Annealing Effect on Hyperparameters

The study finds that learning-rate annealing changes loss and shifts optimal hyperparameters, with effects depending on model size, data horizon, learning rate, and batch size.

  • Motivation: Because decay runs account for roughly 78.6% of total compute in this setup, hyperparameter sweeps over annealed runs impose substantial computational and energy costs.The paper motivates studying constant-learning-rate checkpoints as a possible way to reduce this burden.
  • Loss improvement from LR decay: Annealing-induced loss improvement varies across learning-rate, batch-size, model-size, and data-budget configurations.Figures 7 and 8 show that ΔL is not constant across N, D, η, and b.
  • Loss improvement from LR decay: Larger learning rates and smaller batch sizes generally produce larger loss improvements after decay.The paper relates this pattern to greater optimization noise, while noting that the mechanistic interpretation is framed as a potential explanation.
  • Loss improvement from LR decay: Loss reduction tends to increase at later training horizons and is consistently larger for larger models at matched hyperparameters.These trends extend the dependence of annealing gains beyond the learning-rate and batch-size settings.
  • Optimal hyperparameters shift after LR decay: After decay, optimal configurations favor learning rates up to approximately 4× larger, while smaller batch sizes often lower post-decay loss.The batch-size trend is less consistent and includes exceptions.
  • Optimal hyperparameters shift after LR decay: Stable-phase runs consistently favor the smallest learning rate in the tested grid, indicating that stable and decay phases do not share the same optimum.The observed shift persists across model sizes and token budgets.

5 Scaling Laws for Performance Prediction

The paper evaluates scaling laws for cross-entropy loss across model and data scales, focusing on whether model–data interactions improve prediction beyond separable power laws. The interaction-aware Skaling form fits extreme observations better and extrapolates substantially more accurately to a held-out 1.7B model.

  • Experimental setup: The analysis collects validation-loss measurements at multiple model sizes and data budgets using the lowest-loss hyperparameter configuration for each pair.Raw validation losses are used here to avoid propagating uncertainty from the smoothing fit.
  • Chinchilla law: The separable Chinchilla law fits intermediate model and data scales but underperforms at extremes and extrapolates poorly to the held-out 1.7B model.Residuals indicate that the effects of model size and training data are not fully separable.
  • Chinchilla law: The observed residual interaction implies that increasing data helps more for larger models, while increasing model size helps less when data is scarce.These relationships are reported from the residual patterns of the separable fit.
  • Modeling N-D interaction: The additive interaction model requires G < 0 to capture the observed interaction, conflicting with the desired monotonicity in model size and data.This motivates considering an interaction exponent instead.
  • Modeling N-D interaction: The Skaling law couples model and data capacity through an interaction exponent and provides better fits, particularly at extreme grid observations.The estimated interaction exponent is approximately 0.41.
  • Modeling N-D interaction: RMSE is 0.0056 versus 0.0174 on held-out 1.7B points for the extended model and Chinchilla law, respectively.The paper reports substantially lower test error for the interaction-aware model.
  • Compute-optimal scaling: Compute-optimal loss is defined as the minimum loss across model and data configurations under a fixed compute allocation.The fitted functional form is used to derive a closed-form optimal-loss relationship as a function of compute.

6 Downstream Performance

Downstream error consistently decreases with increasing compute, although the smallest models quickly plateau as additional tokens yield diminishing gains. At matched compute, OpenEuroLLM models achieve lower downstream error than Pythia and SmolLM2 reference models.

  • Downstream performance error consistently decreases as training compute increases across OpenEuroLLM model scales.
  • At the smallest model size, downstream error plateaus quickly because additional training tokens provide diminishing gains.The passage identifies model capacity as the limiting factor in this regime.
  • Larger models continue benefiting from additional compute and follow the compute-performance trend observed for other open-weight models.
  • OpenEuroLLM models achieve lower downstream error than Pythia and SmolLM2 at matched compute where direct comparison is possible.
  • The authors conclude that the training pipeline and hyperparameter optimization generalize across scales and provide a reference for further OpenEuroLLM development.

7 Conclusions and Limitations

The report studies scaling laws for optimal hyperparameters and model performance while quantifying uncertainty from validation sampling, initialization, and data ordering. It finds seed effects can dominate uncertainty at one tested scale, but cautions that extrapolation and transfer across settings remain limited.

  • The study extends scaling-law analysis from jointly optimal learning rates and batch sizes to their marginal behavior, relevant when throughput constrains batch size.
  • Small-scale scaling laws extrapolate successfully over roughly an order of magnitude, but their validity at substantially larger scales remains unclear.
  • Transferability of scaling exponents across datasets, architectures, and training configurations remains limited because reported exponents vary substantially.
  • Validation-loss estimates include randomness from validation sampling, data ordering, and model initialization, with seed effects controlled through random seeds.
  • The validation-loss standard error decreases with model size and token budget, with more performant models producing lower-variance estimates.
  • At 50M parameters and 6B tokens, seed-induced standard deviation is 2.67 · 10^-3 versus 6.3 · 10^-4 from finite validation sampling.The total standard deviation across seeds is 2.7 · 10^-3, and uncertainty is reported as dominated by seed effects at this scale.

A.2 Smoothing Robustness

The robustness analysis compares empirical and smoothed hyperparameter optima across random seeds. Smoothing produces more stable estimates than direct selection, particularly when experiments use a coarse hyperparameter grid.

  • The analysis fits the quadratic loss surface separately across four random seeds for a 50M-6BT model and compares empirical with smoothed optima.
  • Empirical optimal batch size oscillates between 32 and 64 across seeds, while the optimal learning rate remains 0.002.
  • Smoothing is reported as robust for hyperparameter selection and may produce lower-variance estimates than direct empirical selection.
  • The potential advantage of smoothing is especially relevant when the hyperparameter grid is coarse, as in these experiments.

B.1 Scaling Optimal Batch Size with Arbitrary Learning Rate

The study models optimal batch size as a function of learning rate, model size, and data budget using an interaction-aware log-linear form. Larger models, datasets, and learning rates prefer larger batches, while model-size and data-budget effects reinforce one another.

  • Optimal batch size increases with token budget for fixed model size and with model size for fixed data budget.
  • Larger models, larger data budgets, and larger learning rates all prefer larger optimal batch sizes.
  • The learning-rate-by-data-budget interaction is weak, whereas the positive model-size-by-data-budget interaction indicates reinforcing effects on optimal batch size.
  • The three-way interaction term is not statistically significant, so it is removed from the reduced model.
  • The fitted model includes main effects for learning rate, model size, and data budget, plus model-size interactions with data budget and learning rate.
  • The reduced model explains the fitted data with R^2 = 0.981 for the Equation 13 coefficient analysis.
  • Prediction accuracy is tested on held-out 1.7B models across data budgets, using Figure 17 and the corresponding fitted coefficients.

C Fitting Procedure

The fitting procedure estimates scaling-law parameters by minimizing a log-Huber loss, using multiple initializations, unconstrained L-BFGS optimization, and bootstrap resampling for uncertainty.

  • Fitting objective: The log-Huber loss compares measured loss L_i with loss predictions L̂_i from Equation 10 or Equation 11.The fitting procedure uses δ = 10^-3.
  • Optimization: Parameters E, A, and B are optimized in log-space to improve numerical stability.
  • Initialization: The procedure evaluates multiple initialization grids for the Chinchilla and Skaling models before selecting the solution with the lowest final loss.The grids vary log E, log A, log B, α, β, and, for Skaling, κ.
  • Prediction check: Equation 14 predictions are compared with empirical optima and quadratic-slice minima, with pointwise parametric 95% confidence intervals across η and D.The 80BT panel exhibits pronounced noise.
  • Optimization: Unlike the cited prior method, fitting is unconstrained and solved using L-BFGS.
  • Uncertainty: Parameter uncertainty is estimated from 1,000 bootstrap samples, with mean and standard error reported for the resulting estimates.Each bootstrap sample resamples training runs with replacement and repeats the initialization sweep.

D Downstream Evaluation

Downstream performance is measured with the DCLM-CORE suite and compared across OpenEuroLLM and publicly available reference models. The evaluation uses centered task scores, while missing task results constrain coverage for some models.

  • Evaluation suite: DCLM-CORE contains 22 tasks spanning knowledge, reasoning, language understanding, reading comprehension, and symbolic problem solving.Because four task results were unavailable, OpenEuroLLM performance is reported over the remaining 18 tasks.
  • Model-size convention: The model-size convention defines N using Transformer-block and output-head parameters, excluding input embeddings for models with untied embeddings.
  • Evaluation suite: The DCLM-CORE suite is summarized in Table 7 with its tasks and few-shot settings.
  • Metric: Each task score is rescaled, centered on its random baseline, clipped to [0, 1], and averaged across available tasks.This aggregation maps the random baseline to 0 and a perfect score to 1.
  • Reference models: The comparison includes publicly available models spanning multiple sizes and token budgets, including Pythia, SmolLM, Qwen, EuroLLM, and DCLM families.
  • Coverage: Some reference models lack one or two task evaluations, so their downstream scores are averaged over the available tasks.Missing evaluations include SQuAD and, for several models, AGI Eval LSAT-AR.
  • Scaling behavior: OpenEuroLLM models show downstream scaling behavior consistent with existing open-weight models as training compute increases.Figure 18 plots downstream performance against training compute in FLOPs.
Loading 2608.28308v1…