PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, Lei Wu

arXiv:2608.24814v1cs.LG

TL;DR

The paper examines whether norm control provides an independent degree of freedom in language-model loss dynamics. It matches ELR across substantially different LR and norm trajectories, tests practical norm control and scaling-law transfer, and finds that ELR organizes trajectory collapse, cross-method prediction, and delayed acceleration. The resulting law is macroscopic and conditional, with precision depending on normalization design and variation timescale.

  • Problem

    The paper asks whether norm control introduces an independent degree of freedom governing language-model pretraining loss dynamics.

  • Method

    The authors match ELR schedules across runs while varying LR and parameter-norm trajectories, then test practical norm control, scaling-law transfer, and delayed acceleration.

  • Results

    When ELR is matched, loss trajectories collapse across substantially different LR and norm trajectories; ELR-parameterized FSL transfer increases prediction accuracy over LR parameterization, with LR error increasing by 12.9×.

  • Takeaways & Limitations

    ELR is a common coordinate for loss dynamics, while LR scheduling and norm control are mechanisms for realizing an ELR schedule.

  • Takeaways & Limitations

    The claims are limited to loss dynamics, and collapse precision is conditional on normalization design and the timescale of LR–norm variation.

Abstract

from arXiv · show

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

1 Introduction

The paper asks whether norm control adds an independent degree of freedom to loss dynamics and finds that LR and parameter norm primarily matter through their ratio, ELR. Matched ELR schedules produce nearly identical trajectories, while collapse precision is conditional and ELR also supports practical-control transfer and delayed-acceleration analysis.

  • Norm control does not introduce an independent loss-dynamics coordinate within the studied regimes; LR and parameter norm primarily act through their jointly induced ELR schedule.
  • Matched ELR schedules make loss trajectories nearly coincide despite substantially different learning-rate and parameter-norm trajectories.
  • High-precision collapse is conditional on normalization design and sufficiently gradual LR–norm variation rather than implied by exact scale symmetry.
  • ELR collapse extends from prescribed norm trajectories to practical weight decay and Hyperball, with collapse errors of 4.8 × 10−3 and 1.2 × 10−3, respectively.
  • ELR parameterization lets a functional scaling law transfer from non-Hyperball to Hyperball runs, whereas LR parameterization increases prediction error by 12.9×.
  • ELR reshaping captures delayed acceleration, in which norm-controlled training initially has higher loss but later overtakes an uncontrolled baseline.

2 Related Work

Prior work connected norm control and loss dynamics to effective learning rate (ELR), while this paper tests the broader trajectory-level hypothesis systematically and quantitatively.

  • Scale-invariant objectives make ELR the relevant update scale, with η_k/∥W_k∥² for gradient descent and η_k/∥W_k∥ for signSGD and Adam.
  • Weight decay is increasingly viewed as a mechanism regulating parameter norms and effective angular updates, with reported importance for stability and hyperparameter transfer in LLM pretraining.
  • Prior experiments found approximate or late-stage loss alignment when LR or ηλ was adjusted to match ELR-related quantities.
  • The paper extends these observations by testing whether ELR matching aligns full loss trajectories across optimizers, architectures, datasets, and model scales.
  • Matching ELR yields mean trajectory discrepancies of only a few ×10^-3 across diverse training configurations and enables FSL transfer across norm-control regimes.

3 ELR, Norm Control, and Collapse Metrics

The paper defines ELR for Transformer parameters, compares norm-control mechanisms through the ELR schedules they induce, and evaluates trajectory alignment with paired collapse experiments.

  • ELR is defined for matrix-valued Transformer parameters updated with learning rate η_k, providing the schedule used to compare runs.
  • Weight decay shapes ELR implicitly by regulating norm evolution through the shrinkage factor 1−η_kλ, whereas Hyperball fixes each matrix parameter to a sphere of radius R.
  • Hyperball rescales the optimizer update before projection using ˜η_k = η_kR/∥U_k∥_F, with the fixed weight norm determining the corresponding ELR.
  • Collapse experiments pair runs with identical initialization and data order, varying only learning rate or norm control while matching ELR trajectories.
  • Mean collapse error is computed from loss residuals over the training steps included in evaluation.
  • A few ×10^-3 collapse error is below both seed-induced variation in the reference configuration and the 10^-2-scale differences commonly treated as meaningful.

4 The ELR Collapse

ELR-matched runs with substantially different learning-rate and parameter-norm schedules produce nearly identical loss trajectories. Collapse precision is robust but conditional, depending especially on normalization design and the timescale of joint LR–norm variation.

  • The ELR collapse: Different LR and target-norm schedules preserving a common WSD ELR produce nearly coincident loss curves in Llama-124M and Qwen3-MoE experiments.The schedules include WSD, linear-up, linear-down, and sinusoidal learning rates.
  • The ELR collapse: ELR-matched residuals fluctuate around zero without systematic drift, yielding trajectory-level agreement rather than exact pointwise equality.Mean discrepancies are only a few ×10^-3.
  • The ELR collapse: Across 26 ELR-matched comparisons, the median collapse error is 2.5 × 10^-3 and every comparison remains below 5 × 10^-3.Comparable precision also appears for ViTs trained on ImageNet.
  • When Is ELR Collapse Precise?: Removing QK-Norm raises mean collapse error from 2.3 × 10^-3 to 5.2 × 10^-3, while additionally fixing RMSNorm gains raises it to 1.84 × 10^-2.The combined ablation increases error by nearly an order of magnitude.
  • When Is ELR Collapse Precise?: Fixing RMSNorm gains increases collapse error by approximately 3.5× despite making the parameterization more scale invariant.This indicates that static scale invariance alone cannot explain high-precision collapse.
  • When Is ELR Collapse Precise?: Over the tested range, mean collapse error increases from 2.8 × 10^-3 at two modulation cycles to 7.5 × 10^-3 at 32 cycles.Faster modulation produces larger residual oscillations at the imposed frequency while preserving the prescribed ELR.

5 Weight Decay and Hyperball Act through ELR

LR-only interventions show that weight decay and Hyperball alter loss dynamics primarily through the ELR trajectories they induce. Matching those trajectories recovers close loss alignment, apart from a brief Hyperball onset transient.

  • Weight decay: Adapting only the LR of a no-weight-decay run to match an AdamW λ = 0.1 target’s ELR recovers the target loss with collapse error 4.8 × 10^-3.Under the original LR, removing weight decay changes norm evolution and substantially deviates from the target.
  • Hyperball: Adapting only MuonW’s LR to match MuonH’s ELR brings their losses into close alignment after a brief initial transient, with mean error 1.2 × 10^-3 from step 7.5k onward.The transient occurs when the adapted LR is largest.
  • Cross-method conclusion: Reverse interventions, with each norm-control setting serving as the target in turn, yield the same conclusion.This supports the direction-independent character of the intervention result.
  • Cross-method conclusion: Weight decay and Hyperball affect loss dynamics primarily through the ELR trajectories they induce, reducing discrepancies between mechanisms to the 10^-3 scale.Norm control does not necessarily improve convergence; its effect depends on the ELR trajectory induced.

6 ELR Enables Scaling-Law Transfer

The paper replaces LR with ELR in a functional scaling law and tests transfer across norm-control methods without refitting. ELR parameterization remains accurate on unseen Hyperball runs, whereas LR parameterization develops substantial systematic bias.

  • Functional scaling laws: The original lr-FSL models loss dynamics from LR schedules through intrinsic training time and accumulated optimization noise.Increasing LR advances intrinsic time while increasing noise injection.
  • Functional scaling laws: The elr-FSL replaces LR with effective learning rate throughout intrinsic-time and noise terms while leaving the functional form unchanged.The two formulations differ only in whether dynamics are parameterized by LR or ELR.
  • Transfer across norm-control methods: Both FSL variants are fitted on the same four non-Hyperball trajectories and evaluated without refitting on held-out ID and unseen Hyperball runs.The experiment uses Llama-124M on FineWeb under no decay, weight decay, and Hyperball.
  • Transfer across norm-control methods: On unseen Hyperball runs, elr-FSL reaches mean RMSE 0.0212 versus 0.2508 for lr-FSL, a 11.83× larger error for the LR formulation.ELR also reduces held-out ID mean RMSE from 0.0239 to 0.0133.
  • Transfer across norm-control methods: ELR parameterization enables a scaling law fitted under one set of norm-control methods to transfer to another.The result extends beyond pairwise trajectory alignment to predictive scaling-law transfer.

7 Delayed Acceleration: Explanation and Control

Norm control affects delayed acceleration through the ELR schedule it induces. Weight decay restrains norm growth and preserves ELR early, while reshaping ELR can preserve the late-stage benefit and lower final loss.

  • Delayed acceleration: Weight decay initially raises loss but later overtakes the unregularized baseline, while ELR-guided norm control further lowers final loss.These experiments use Llama-124M trained on FineWeb with AdamW.
  • ELR mechanism: Without weight decay, faster norm growth makes ELR decay substantially faster than the nominal LR.Weight decay restrains norm growth and sustains a larger ELR for longer despite the shared LR schedule.
  • Why the gain is delayed: Delayed acceleration reflects a temporal separation: larger early ELR acquires a learning advantage, while accumulated noise initially masks it and later diminishes.As late ELR decays, new noise injection falls while previously accumulated noise is forgotten, revealing the earlier advantage.
  • Controlling delayed acceleration: A prescribed norm schedule that follows weight decay early and grows faster later produces similar early ELR, smaller late ELR, and preserves delayed acceleration.The reshaped schedule also attains a lower final loss under the shared LR schedule.
  • Generalization: Analogous delayed acceleration under Hyperball is explained through the same ELR-based mechanism.

8 Conclusion and Discussion

The paper presents ELR as the common coordinate through which learning-rate scheduling and norm control shape pretraining loss dynamics. It also defines the empirical conditions and scope of this reduction, emphasizing that ELR is a design target rather than an exact consequence of scale symmetry.

  • LR and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR).
  • ELR collapse is not explained by exact scale symmetry because transformer architectures contain scale-dependent residual, projection, activation, embedding, and output pathways.
  • Designing the ELR schedule first separates the desired effective trajectory from the LR–norm mechanism used to realize it.
  • LR scheduling and norm control can be treated as mechanisms for realizing a common ELR schedule rather than independent dynamical coordinates.
  • The ELR coordinate is defined using matrix-valued parameter norms, while vector-valued parameters can retain scale dependence in the loss.

A.2 Interpreting Collapse Errors at the 10−3 Scale

The paper compares ELR-collapse errors with ordinary run-to-run stochastic variation using the same trajectory-discrepancy measure. In a representative Llama-124M configuration, seed-induced variation is substantially larger than ELR-collapse error.

  • ELR collapse: 1.8 × 10−3 to 2.6 × 10−3 is the collapse-error range for three ELR-matched Llama-124M runs with both seeds fixed.The same mean absolute trajectory discrepancy is used for collapse errors and stochastic variation.
  • Comparison: More than four times smaller is the largest ELR-collapse error than the lower stochastic-variation baseline.Across the reported values, the stochastic-variation difference is a factor of 4.3–9.0.
  • Stochastic variation: 1.11 × 10−2 and 1.62 × 10−2 are the time-averaged loss variations from initialization and data-order seeds, respectively.These values are computed by averaging pairwise mean absolute discrepancies across repeated runs.

B.1 Training Configurations

The appendices define the models, datasets, optimizers, smoothing, and ELR-matching protocols used for prescribed-ELR validation. These experiments vary LR and parameter-norm schedules while evaluating trajectory-level collapse errors across configurations.

  • Models and optimizers: The study includes Llama, Qwen3, Kimi Delta Attention, and multiple optimizers, including AdamW, Muon, and Signum.Additional validation spans Qwen3-145M, Kimi Delta Attention, and Llama-124M experiments.
  • Datasets: FineWeb and C4 are among the pretraining datasets used in the validation experiments, with OpenWebText included in additional Qwen3-145M validation.The appendix identifies FineWeb, C4, and OpenWebText as experimental datasets.
  • ELR definitions: ELR conversion depends on optimizer update parameterization, with a distinct conversion for Hyperball because it normalizes the update.The protocols use optimizer-specific ELR definitions for matching prescribed schedules and reference-run schedules.
  • Matching protocols: The two matching protocols either prescribe LR and norm schedules jointly or adapt only the comparison run’s LR to reproduce a reference ELR.Reference-run matching leaves the comparison optimizer and norm-control mechanism unchanged.
  • ELR-matching protocol: Three LR schedules—linear-up, linear-down, and sinusoidal—are paired with adjusted norm schedules to realize the same prescribed ELR.The mean collapse error ∆coll is evaluated over post-warmup training steps using smoothed losses.

D Weight Decay and Hyperball (Supplement to Section 5)

LR-only ELR matching aligns losses across weight-decay and Hyperball interventions, showing that practical norm-control methods can be compared through the ELR schedules they induce. The fitted ELR-based scaling law also transfers more accurately across held-out norm-control trajectories.

  • Weight decay: The loss trajectories remain closely aligned when AdamW matching is performed in either direction between λ = 0 and λ = 0.1.The λ = 0.1 run is the target in one direction, and the λ = 0 run is the target in the reverse direction.
  • Hyperball: 2.5 × 10−3 is the mean collapse error when matching MuonW and MuonH in both directions.MuonW uses weight decay, whereas MuonH uses Hyperball.
  • Normalization sensitivity: Fixing RMSNorm gains increases the LR-only intervention collapse error to the 10−2 scale.This reproduces the normalization sensitivity seen in prescribed-ELR experiments even when parameter norms evolve freely.
  • Scaling-law transfer: 5.61× for cosine and 19.96× for WSD-10% are the Hyperball improvements achieved by elr-FSL over lr-FSL.Across all evaluated trajectories, elr-FSL has lower RMSE on nine of ten trajectories.

F.1 Acquired Early, Revealed Late: A Controlled WSD Test

The controlled WSD experiment tests whether norm-control benefits can be acquired early but become visible only after a sufficiently long low-ELR decay phase. The same delayed-acceleration pattern also appears under Hyperball.

  • Controlled WSD test: 0.1 and 0.3 are the WSD decay ratios used to vary the duration of the terminal decay phase.Peak LR and training budget are held fixed while the decay window changes.
  • Short decay: With a 0.1 decay ratio, the λ = 0.1 run remains above the unregularized baseline at the end of training.The terminal decay phase is too short to reveal the signal-learning advantage.
  • Long decay: With a 0.3 decay ratio, the λ = 0.1 run overtakes λ = 0 during decay and reaches a lower final loss.The gain is accumulated before the crossover but becomes observable only during a sufficiently long decay phase.
  • Hyperball comparison: Under Hyperball, MuonH initially has higher loss than MuonW but later overtakes it and reaches lower late-stage loss.Both runs use linear schedules, and the pattern is described as delayed acceleration.
  • ELR interpretation: Hyperball’s slower ELR decay accelerates signal learning while increasing noise accumulation, initially masking the advantage until terminal decay reveals it.The same mechanism is associated with a 1.2×10−3 mean collapse error when MuonH’s normalized update coefficient is matched.
Loading 2608.24814v1…