PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Hasan Amin, Wei-Kai Chang, Rajiv Khanna

arXiv:2609.09116v1cs.LG

TL;DR

Scale invariance makes the effective step depend on the evolving parameter norm, but the per-step interaction with schedules and weight decay has remained incompletely characterized. This paper derives an exact scalar schedule law and extends it to stochastic optimization and multiple optimizer classes. In a solved model the balance point is unstable, and experiments show that controlling the scalar can organize training behavior.

  • Problem

    The exact per-step law governing the feedback between schedules, weight decay, and effective steps in scale-invariant optimization remains unresolved.

    Source: §1 · p. 2

  • Method

    The paper derives an exact recurrence separating scalar schedule forcing from norm-driven self-quenching, then analyzes a solved normalized-linear model and extends the framework to stochastic optimization and homogeneous optimizers.

    Source: §1 · p. 2

  • Results

    The solved model’s interior balance point is an unstable spiral source, and the recurrence holds with high precision across dynamical systems and neural networks while B_t control yields performance peaking near the predicted boundary.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The scalar B_t provides a coordinate for interpreting and controlling scale-invariant training dynamics under schedule changes.

    Source: §1 · p. 2, §6 · p. 10

    The exact schedule law relies on positive scale invariance and an update direction that scales with parameter norm as specified by the framework.

    Source: §4 · p. 7

Abstract

from arXiv · show

Normalization renders large parts of neural networks effectively scale invariant, inducing a hidden feedback loop in which learning-rate schedules and weight decay interact through the parameter norm to control the effective step taken by the optimizer. We show that this interaction is governed by an exact discrete-time law: a single scalar quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect. This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes. To understand the underlying mechanism, we provide exact analysis of a fully solved normalized regression model where the dynamics reduce to two dimensions and show that the balance point is intrinsically unstable, implying that constant learning rate with weight decay cannot stably maintain an interior equilibrium and instead produces recurrent behavior driven by discrete-time Jacobian structure. We further extend this perspective across optimizers through unified homogeneous-optimizer framework that reveals a structural dichotomy in self-quenching strength, providing a first-principles explanation for why adaptive methods exhibit systematically weaker stabilization under normalization. Across dynamical systems and neural networks (MLP, CNN, GPT2 / MNIST, CIFAR, wikiText, OpenWebText), the predicted law holds with high precision and enables direct control of training via the identified scalar, with performance peaking sharply at the predicted boundary. Together, these results isolate a single governing quantity for scale-invariant optimization, providing a precise and actionable lens on training dynamics, optimizer behavior, and schedule design in modern deep learning. Code is available in https://github.com/shasanamin/normalized-optimization-dynamics.

1 Introduction

The paper derives an exact schedule law for scale-invariant optimization, separating schedule forcing from norm-driven self-quenching. It connects that law to unstable recurrent dynamics, optimizer differences, and empirical control of training.

1 Introduction

  • An exact discrete-time law reduces schedule forcing in scale-invariant blocks to a scalar B_t, whose competition with geometric self-quenching sets a sharp contraction/expansion boundary.The law applies under arbitrary time-varying schedules and requires only scale invariance.Source: §1 · p. 2
  • In a solved normalized-linear model, the interior balance point is an unstable spiral source, so constant learning rate with weight decay produces recurrent rather than stable interior behavior.The model also has an exact period-two orbit on the sphere.Source: §1 · pp. 2–3
  • A homogeneous-optimizer framework finds quadratic self-quenching for SGD and SGDM but only linear self-quenching for Adam, explaining adaptive methods’ greater expansion tendency under normalization.The Adam comparison is stated for the ε → 0 limit.Source: §1 · pp. 2–3
  • Across dynamical systems and neural networks, the recurrence holds with high precision, and schedules targeting B_t produce a sharply peaked performance curve near the predicted boundary.The identified scalar is presented as a coordinate for controlling training dynamics.Source: §1 · pp. 2–3
  • The stochastic extension shows that minibatch noise can accumulate into deviations from expected dynamics and shift the contraction/expansion boundary according to noise variance.The stochastic analysis characterizes both the expected effective-stepsize evolution and the accumulated deviations.Source: §1 · pp. 2–3

2 Exact schedule law

The exact schedule law separates schedule forcing from geometric self-quenching, yielding a per-step contraction test for scale-invariant blocks. Its stochastic extension replaces the deterministic boundary with an expected drift threshold and accounts for accumulated minibatch noise.

2.1 Deterministic exact schedule law for scale-invariant blocks

  • For scale-invariant blocks updated by SGD with coupled weight decay, the exact recurrence separates schedule forcing B_t from self-quenching caused by norm growth.The derivation uses the polar decomposition of the parameter block and holds under arbitrary schedules.Source: §2.1 · pp. 3–4
  • When B_t ≤ 1, the effective stepsize contracts unconditionally; when B_t > 1, contraction requires Φ_t∥ḡ_t∥ ≥ √(B_t − 1).The switching surface links this local contraction criterion to balance in the solved model.Source: §2.1 · p. 4
  • The factorization B_t = (η_t+1/η_t)/(a_ta_t+1) maps standard schedules to scalar trajectories: step decay gives contraction shocks, while cosine decay sweeps across the boundary.Constant learning rate with weight decay instead gives B_t > 1 and sustained expansion pressure.Source: §2.1 · p. 4

2.2 Stochastic extension of exact schedule law

  • Under minibatching, the expected log effective-step drift is schedule forcing log B_t minus the expected geometric-quenching term, with switching at log B_t = q_t.The stochastic recurrence retains the same forcing-versus-quenching structure.Source: §2.2 · p. 4
  • Stochasticity enters cumulative effective-step evolution as a zero-mean martingale around the predictable schedule–quenching drift.The stated finite-horizon concentration result assumes conditionally sub-Gaussian increments.Source: §2.2 · p. 4
  • The near-critical stochastic band depends on state, signal, variance, and batch size, and systematic deviations can accumulate over training.The law can also be inverted to choose B_t for a desired local drift after the ordinary backward pass.Source: §2.2 · p. 5

3 A solved normalized-linear model

The solved normalized-linear model provides finite-horizon guarantees and an exact two-dimensional reduction under isotropic covariance. In that reduction, the balance point is unstable, while decaying schedules can restore contraction.

3 A solved normalized-linear model

  • On the positive hemisphere, the model supplies a schedule-aware finite-horizon loss certificate whose guarantee depends explicitly on the learning-rate and weight-decay schedules.With constant positive weight decay, the radius stabilizes and the certificate saturates at a positive floor.Source: §3 · pp. 5–6

3.1 Exact isotropic reduction and spiral instability

  • With isotropic covariance and a constant schedule, the full dynamics reduce exactly to a two-dimensional map in alignment and effective stepsize.The recurrence links alignment progress to the same forcing–self-quenching mechanism as the general schedule law.Source: §3.1 · p. 6
  • The unique nontrivial fixed point lies on the switching surface, and the sphere dynamics also contain an exact period-two orbit.The switching-surface balance in the reduced model is the same condition as the local schedule law.Source: §3.1 · p. 6
  • The fixed point is an unstable spiral source, so perturbations grow while rotating and the system cannot stably maintain an interior equilibrium.Under decaying schedules, B_t can fall below 1, making the effective stepsize decrease monotonically and suppressing this instability in the model.Source: §3.1 · pp. 6–7

4 Extension beyond SGD through homogeneous-optimizer template

A homogeneous-optimizer framework extends the schedule-law perspective beyond SGD by classifying update directions according to their scaling with parameter norm. The resulting exponent differentiates self-quenching across SGD, momentum, and Adam.

4 Extension beyond SGD through homogeneous-optimizer template

  • The homogeneous-optimizer framework extends exact schedule laws to updates whose scale-free state evolves in closed form, classifying them by the norm-scaling degree ν of their update direction.The template relies on polar decomposition and an update direction scaling as r^-ν under positive scale invariance.Source: §4 · p. 7
  • The denominator exponent 1 + ν quantifies self-quenching strength; SGD and SGDM have ν = 1, while Adam has ν = 0.Larger ν means stronger suppression of the effective stepsize through norm growth.Source: Then the dynamics satisfy · p. 7

4.1 SGD with momentum (SGDM): exact augmented-state law and radial amplification

  • For SGDM, momentum history prevents closure in the effective stepsize alone, but closure is restored by augmenting the state with one scale-free variable.The augmented state includes z_t = r_t m_t+1.Source: §4.1 · p. 7
  • A positive radial momentum component reduces the recurrence denominator and increases the next effective stepsize, creating an amplification channel absent from plain SGD.This is an optimizer-specific mechanism for stronger expansion dynamics.Source: §4.1 · p. 8

4.2 Adam: exact closure via normalized moments

  • For Adam at ε = 0, closure uses scale-free moment variables and its normalized update direction has homogeneity degree ν = 0.The exact law is stated for the ε = 0 case.Source: §4.2 · p. 8
  • Adam’s denominator exponent of 1 gives only linear self-quenching, while preconditioning can add a radial amplification channel and make adaptive methods more expansion-prone under normalization.For ε > 0, the additive denominator term produces a smooth deviation from the exact scale-invariant law.Source: §4.2 · p. 8

5 Experiments

Across exact maps and neural networks, the experiments support a one-scalar account in which Bt organizes expansion and contraction, while optimizer-specific self-quenching shapes the dynamics. Directly setting Bt also produces predictable changes in training behavior.

5 Experiments

  • Across exact maps, neural networks, and optimizer variants, results support Bt as the scalar governing expansion and contraction, with optimizer-specific self-quenching shaping dynamics.The experiments track expansion fraction and recurrence residual as diagnostics.Source: §5 · p. 8

5.1 Exact-map validation: the law is exact and predictive

  • The exact-map recurrence residual stays below 2 × 10−15; constant, step-decay, and cosine schedules produce the predicted sustained, abrupt, and gradual regime changes.Cosine decay moves Bt smoothly across the expansion–contraction boundary.Source: §5.1 · p. 8, Figure 1 · p. 9

5.2 Neural validation: Bt organizes expansion across models

  • On BN MLP and ConvNet models, expansion tracks Bt across schedules, with transitions at Bt = 1 and recurrence residuals near 10−6.These residuals are reported at float32 precision.Source: §5.2 · p. 9

5.3 Target-Bt intervention: isolating causality

  • On CIFAR-10, accuracy peaks at B = 1; below it the effective stepsize collapses, while above it expansion persists but training destabilizes.The authors report that small perturbations rapidly degrade performance and that directly controlling Bt determines training behavior.Source: §5.3 · p. 9, Figure 2 · p. 9

5.4 Optimizer mechanisms: validating the homogeneous law

  • SGDM’s radial amplification matches the augmented-state law, Adam approaches its ε = 0 recurrence as ε → 0, and denominator exponents separate SGD/SGDM (ν = 1) from Adam variants (ν = 0).For SGDM, the contraction condition matches observed behavior on > 99.998% of steps; empirical expansion rates are 0.994 for ct > 0 and 0.56 for ct ≤ 0.Source: §5.4 · p. 10, Figure 3 · p. 10

5.5 Architecture stress test: LayerNorm transformers on language modeling

  • In LayerNorm language models, the Bt identity and ν-classification persist across WikiText and OpenWebText, supporting a geometry-based account beyond image architectures.For SGD/SGDM, constant and step schedules sustain expansion fractions of 0.999–1.000, while cosine drives them to ∼10−4 after warmup.Source: §5.5 · p. 10

6 Discussion

The discussion positions the paper’s exact per-step law as a sharper account of scale-invariant optimization dynamics. It identifies norm-driven effective-step boundaries as complementary to curvature-based stability analyses and notes decoupled decay as a future direction.

6 Discussion

  • The paper analyzes scale-invariant optimization and reports theoretical results on self-quenching, a solved normalized-linear model, and a unified homogeneous-optimizer framework.The authors identify generalizing the analysis to decoupled-decay optimizers, which lack an explicit closed-form solution, as future work.Source: §6 · p. 10

A Related Work

  • Prior work studied scale-invariant feedback through equilibrium, mixing, periodic destabilization, and regime transitions; this work isolates an exact per-step, per-block law for arbitrary schedules.The law factors schedule forcing and geometric self-quenching through Bt.Source: A Related Work · p. 14
  • Unlike curvature-driven edge-of-stability analyses, the paper identifies a norm-driven boundary for contraction and expansion that yields a spiral-source instability in a solved model.The authors describe this as a complementary dynamical account of oscillatory or near-critical behavior.Source: A Related Work · p. 14

B Notation and organization of proofs

The appendix maps the notation and proof organization, then establishes scale-invariance facts used to derive the feedback law, descent and PL results, and a finite-horizon certificate.

C.5 Proof of Theorem 3.2

  • Theorem 3.2’s proof constructs a one-step r⁴ recursion, unrolls it over the horizon, and applies the PL inequality for a schedule-aware finite-horizon certificate.The proof concludes with the theorem’s stated alternative involving the terminal effective stepsize or an earlier condition.Source: B Notation and organization of proofs · p. 14, §C.5 · p. 19

B Notation and organization of proofs

  • Table 1 collects main-text and appendix notation, while the proof roadmap lists the auxiliary results and theorem proofs in Section C.The roadmap includes the polar-dynamics feedback law, descent and PL arguments, the schedule-aware finite-horizon certificate, and optimizer-specific proofs.Source: B Notation and organization of proofs · p. 14, Table 1 · p. 15

C.1 Auxiliary scale-invariance facts used in Theorem 2.1

  • Scale invariance makes the scale-free gradient independent of radius and tangent to the sphere, while the Hessian is homogeneous of degree −2.The Hessian homogeneity is used in the ambient descent proof.Source: §C.1 · pp. 15–16

C.2 Proof of Theorem 2.1: polar dynamics, feedback law, and threshold

  • For the stated iteration, the proof establishes that if Bt ≤ 1, then Bt ≤ 1 ≤ 1 + Φt²∥¯gt∥², and consequently the potential cannot increase: Φt+1 ≤ Φt always.Source: §C.2 · p. 16

C.3 Proof of the descent statement in Theorem 3.1

  • The descent proof uses a Hessian bound and shows the relevant multiplicative factor is nonnegative when Φt ≤ 2/Lamb.This condition establishes the stated descent result.Source: §C.3 · p. 17

C.4 Proof of the PL part of Theorem 3.1

  • Under the stated normalization and realizable-target assumptions, the PL proof obtains a lower bound on the scale-free gradient norm.The bound is ∥ḡ(u)∥² ≥ m σ(u)² (1 − q(u)²).Source: §C.4 · pp. 18–19

C.6 Proof of Theorem 3.4, Proposition 3.5, and Proposition 3.6

The proof establishes that the normalized-linear model’s interior fixed point is an unstable spiral source, and develops stochastic and optimizer-specific extensions of the effective-step analysis.

C.7 Direct verification of the period-2 lift in Proposition 3.5

  • The two lifted period-2 points project to the same reduced fixed point, and direct substitution verifies that the direction update swaps them.Replacing e by −e gives the reverse transition.Source: §C.6 · p. 21, §C.7 · p. 24

C.6 Proof of Theorem 3.4, Proposition 3.5, and Proposition 3.6

  • For every a in (0, 1), the fixed point has complex-conjugate eigenvalues with modulus greater than 1, making it an unstable spiral source.The maximum eigenvalue modulus occurs at a = 1/2, where its square is 5/4.Source: §C.6 · p. 23

C.9 Proof of Theorem 4.2

  • The effective-step proof uses the radius update, normalization, and an orthogonal decomposition to obtain a recurrence and threshold condition.The supplied proof passages also derive the direction update and its relation to z_t.Source: §C.9 · p. 24

C.10 Proof of Theorem 4.3

  • Theorem 4.3’s proof obtains the second-moment update by squaring the scale factor and identifies the radius-degree-zero update.The resulting update matches Theorem 4.1 with ν = 0.Source: §C.10 · p. 24

C.8 Proof of Theorem 4.1 (homogeneous-optimizer template)

  • The optimizer template identifies Adam’s update direction as degree 0 in the current radius, matching the ν = 0 case.The update matches Theorem 4.1 and yields the stated radius ratio.Source: §C.10 · p. 24

C.11 Proof of Theorem 2.3

  • The stochastic proof decomposes accumulated log-step changes into a martingale difference and establishes a finite-horizon concentration bound.The argument uses conditional centering and a deterministic variance budget.Source: §C.11 · p. 25

C.12 Additional corollary: Minibatch near-critical band

  • With minibatch noise, annealing the forcing lowers the noise-supported effective-step scale, and log B_t ≤ 0 gives strictly negative conditional log drift when q_t > 0.The moving critical band depends jointly on gradient signal and minibatch variance; conditional i.i.d. sampling is used for the explicit variance reduction.Source: §C.12 · pp. 25–26

C.12.1 Assumption and claim audit

  • The stochastic result characterizes the effective-step state, not general convergence or generalization; the sharp perturbative band additionally requires X_t ≤ ρ < 1.The experiments described here use synthetic, MNIST, and CIFAR-10 models in controlled settings, with language-model experiments on WikiText and OpenWebText.Source: §C.12.1 · p. 26, D Experimental details · p. 27

D.1 Training setups for Linear/MLP/ConvNet on Synthetic/MNIST/CIFAR-10

The experiments span synthetic and vision models as well as GPT-2-style language models, while tracking scale-invariant blocks and optimizer-specific statistics. Additional tests report spiral instability, robustness to minibatch noise and anisotropy, and continuity as Adam’s ε approaches zero.

D.1 Training setups for Linear/MLP/ConvNet on Synthetic/MNIST/CIFAR-10

  • Controlled Bt experiments used an MNIST BN MLP for T=120 steps (batch size 128, λ=0.05) and a CIFAR-10 ConvNet for 10,000 steps (λ=0.05), enforcing Bt≡B with η0=0.5.Source: §D.1 · p. 27

D.2 Training setups small_gpt2/gpt2 on WikiText/OpenWebText

  • The language-model tests use GPT-2-style transformers with affine-free LayerNorm and track attention and projection blocks treated as scale invariant.The setup tracks 8 blocks in small_gpt2 and 24 in gpt2, while excluding embeddings, MLP sublayers, and the LM head.Source: §D.2 · p. 27

D.3 Tracked statistic

  • The tracking protocol records weights, minibatch gradients, optimizer-specific directions, post-update weights, and scale and expansion statistics.For Adam, it logs both the preconditioned direction and an ε = 0 reference direction derived from the same moments.Source: §D.3 · pp. 27–28

E Additional experiments results

  • The additional experiments are grouped into theoretical validation, controlled ablations on MNIST and CIFAR-10, and language-dataset results.The validation includes the spiral-source, minibatch and anisotropy robustness, and Adam ε-continuity tests.Source: E Additional experiments results · p. 28, §E.1 · p. 28

E.1 Additional validation of the theoretical predictions

  • The validation reports outward spiraling near the fixed point, persistent recurrence across batch sizes and condition numbers, and smooth convergence of Adam’s recurrence as ε approaches zero.The Jacobian matches finite-difference estimates to ∼10^-8.Source: §E.1 · p. 28

E.2 Supporting ablations and controls

The supporting controls connect training behavior to B_t across weight-decay, optimizer, and schedule variations. The reported phase diagram links time spent in the subcritical regime with expansion suppression.

E.2 Supporting ablations and controls

  • With the learning-rate path fixed, accuracy decreases monotonically as weight decay increases, alongside a larger effective expansion floor and less time in subcritical regimes.The near-critical controls indicate that performance is tied to criticality rather than simply to λ = 0.Source: §E.2 · p. 28
  • The minibatch robustness figure reports persistent expansion dynamics and floating-point-precision residuals across batch sizes.Source: Figure 5 · p. 29
  • The anisotropy robustness figure reports that the recurrence remains exact while trajectories deform with condition number.Source: Figure 6 · p. 29
  • Across SGD, SGDM, and Adam variants, the same regime structure appears, with differences attributed to the homogeneity exponent; transitions are consistent across network depth.The passage identifies B_t as acting uniformly across scale-invariant blocks.Source: §E.2 · p. 29
  • In the cosine-family sweep, the fraction of time spent subcritical predicts expansion suppression with correlation 0.99.The sweep varies λ and η_min.Source: §E.2 · p. 29

E.3 Language Modeling Experiments

The language-model experiments test the recurrence across optimizers, schedules, and two GPT-2-style models. They report near-float32 residuals and schedule-dependent expansion patterns for SGD and SGDM, with a distinct, weaker self-quenching profile for Adam.

E.3 Language Modeling Experiments

  • For SGD and SGDM, the median logged ratio residual is 1.19 × 10^-7 on every schedule and both models, with worst-case median below 6 × 10^-7.The results cover tracked attention blocks and seeds.Source: §E.3 · p. 29
  • For SGD and SGDM, constant and step schedules yield expansion fractions of 0.999–1.000, while cosine brings the fraction to about 10^-4 after warmup.For SGD, the logged threshold accuracy exceeds 0.99999.Source: §E.3 · p. 30
  • Coupled Adam’s ratio residuals were 7.2 × 10−6–6.2 × 10−5 on WikiText and 7.6 × 10−6–1.9 × 10−4 on OpenWebText; expansion was 0.44–0.49 under constant/step and 0.033/0.095 under cosine, respectively.Source: §E.3 · p. 30
  • The language-model validation frames its central finding as the survival of blockwise schedule diagnostics from BN-Conv vision models to LayerNorm-attention language models.Source: §E.3 · p. 30

F Further discussion on the scope, limitation, and future directions.

The discussion connects the exact scalar recurrence to instability, optimizer differences, and schedule control, while clarifying where the results do and do not apply. It also motivates the effective step as a measure of directional motion under scale invariance.

F Further discussion on the scope, limitation, and future directions.

  • The exact recurrence separates schedule and decay forcing from geometric self-quenching, yielding a sharp contraction–expansion boundary without approximations or model-specific assumptions beyond scale invariance.The boundary is at Φ∥¯g∥ = B^-1.Source: F Further discussion on the scope,… · p. 33
  • In the solved isotropic model, the interior balance point is an unstable spiral source on that boundary, but global convergence to a limit cycle is not proved.The analysis also does not extend the closed-form result to anisotropic covariance or general deep networks.Source: F Further discussion on the scope,… · p. 33
  • The homogeneity exponent determines optimizer self-quenching strength: it is quadratic for SGD/SGDM and linear for Adam, explaining adaptive methods’ stronger expansion tendencies.Source: F Further discussion on the scope,… · p. 33
  • The exact result is block-conditional: each covered block must satisfy positive scale invariance with other state fixed, and GPT experiments do not establish theorem coverage of every Transformer component.Canonical covered blocks are weights immediately upstream of exact normalization when no additive or bypass path breaks the symmetry.Source: F Further discussion on the scope,… · p. 33
  • For Adam, the exact optimizer-level result uses external multiplicative shrinkage; L2 regularization inside Adam’s moments is outside it, and no universal B=1 peak is claimed for standard AdamW Transformers.Finite ε is treated as a smooth perturbation, and Adam preconditioning can add a radial component.Source: F Further discussion on the scope,… · p. 33
  • Controlling Bt provides a schedule-design coordinate: enforcing its trajectory yields peak performance at the instability boundary B=1.This lets schedules be designed around the contraction–expansion boundary rather than by tuning learning rate and weight decay directly.Source: F Further discussion on the scope,… · p. 33
  • The one-scalar factorization relies on coupled weight decay; decoupled decay breaks it, while anisotropic and deep-network settings retain the recurrence but not the full closed-form dynamics.For Adam, ε=0 is the limiting law and ε > 0 is a smooth perturbation.Source: F Further discussion on the scope,… · p. 33

G Motivation and Intuition

  • For scale-invariant losses, Φt controls directional motion on the unit sphere and is monotonically related to geodesic distance; norm growth reduces the effective step through self-quenching.Effective-step expansion or contraction refers to Φt+1 ≷ Φt, not growth or shrinkage of the raw parameter norm.Source: G Motivation and Intuition · pp. 34–35

Related papers

Loading 2609.09116v1…