PaperlayerView exact revision on arXiv

Source-linked AI summary

The Drift Contract: Spectral Updates for Depth-Robust Local Learning

Fabien Polly

arXiv:2609.26811v1cs.LG

TL;DR

Local learning makes layer updates structurally parallel but has faced depth-related accuracy degradation and brittle hyperparameters. The paper applies Muon-style spectral update geometry to per-layer local updates and scales the learning rate by input RMS to form a drift contract. Local Adam drops 44.4 to 31.3 percent over depths 12–48 despite re-tuning; spectral updates lose 3.0 percentage points unchanged; 3e-3 is best in tested grids spanning 16× width and 4× depth.

  • Problem

    Local learning makes layer updates structurally parallel but has faced depth-related accuracy degradation and brittle hyperparameters.

    Source: §1 · p. 1

  • Method

    The paper applies Muon-style spectral update geometry to per-layer local updates and scales the learning rate by input RMS to form a drift contract.

    Source: §1 · p. 2

  • Results

    Local Adam drops 44.4 to 31.3 percent over depths 12–48 despite re-tuning; spectral updates lose 3.0 percentage points unchanged; 3e-3 is best in tested grids spanning 16× width and 4× depth.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The drift-contract reading makes the step size interpretable and provides a per-layer, input-conditioned bound on weight-induced output change per step.

    Source: §7 · p. 7

    In the synthetic stability control, RMSNorm plus decoupled weight decay shifts the spectral stability advantage to global Muon.

    Source: §4.4 · p. 5

Abstract

from arXiv · show

Local learning trains each layer with its own auxiliary loss and no global backward pass, which makes layer updates structurally parallel. Two problems have kept it marginal: accuracy degrades as depth grows, and hyperparameters are fragile. We apply Muon-style spectral update geometry (momentum orthogonalization with spectral step scaling) to per-layer local updates, an intersection not previously studied. On CIFAR-10 MLP benchmarks with local linear heads, a single step-size setting is the best value in our tested grids from width 128 to 2048 and from depth 12 to 48, while local Adam requires re-tuning along both axes and still collapses at depth 48 (31.3 percent re-tuned per depth, 19 percent with its depth-12 setting transferred, vs 42.7 percent for the spectral update at its unchanged setting). At five seeds and width 512 the spectral update leads local Adam by a clear margin (48.9 +/- 0.5 vs 46.6 +/- 0.3). Prospectively specified controls attribute the transfer and most of the depth robustness to the spectral geometry itself rather than to any step-size rule on top of it. We additionally formulate the step size as a drift contract, lr = epsilon / RMS(input), which bounds each layer's weight-induced pre-activation change per step, conditioned on its current input. The contract yields a small gain over the best fixed learning rate where that baseline is measured, makes the step size interpretable, and provides a per-layer, input-conditioned drift bound that standard optimizers do not offer. We report one negative result: with RMSNorm and weight decay in the trunk, the stability benefit of spectral updates accrues to global rather than local training, so the local advantage concentrates precisely where normalization is absent.

1 Introduction

The paper studies spectral update geometry for local learning, which avoids a global backward pass but has faced depth degradation and brittle hyperparameters. It also introduces an input-conditioned drift contract for layer updates.

1 Introduction

  • Local learning updates layers from auxiliary-head gradients without a global backward pass, enabling structural parallelism but historically suffering depth pathology and brittle hyperparameters.The pipeline implementation reports 4.34x throughput at 12 layers.Source: §1 · p. 1
  • The paper’s first systematic study tests whether Muon-style spectral update geometry, previously grounded in global gradients, applies to local gradients.The study uses a prospectively specified protocol and controls.Source: §1 · pp. 1–2
  • Local Adam falls from 44.4 to 31.3 percent as depth grows from 12 to 48 despite re-tuning, while spectral updates lose 3.0 percentage points with one unchanged setting.The depth result compares local Adam with per-depth re-tuning against spectral updates using a single setting.Source: §1 · p. 2
  • One setting, 3e-3, is the best-performing value in tested grids across a 16x width range and a 4x depth range.The optimum is interior and occurs at the same value across both axes.Source: §1 · p. 2
  • The drift contract sets lr = ϵ/RMS(input), bounding each layer’s weight-induced functional change per step conditional on its current input.It edges the best fixed learning rate where both are measured and beats local Adam at every reported width.Source: §1 · p. 2
  • With RMSNorm and weight decay in the trunk, the stability advantage shifts to global rather than local training.The introduction also attributes width transfer to spectral scaling, not the drift contract.Source: §1 · p. 2

2 Method

Each layer of the depth-L MLP minimizes its own cross-entropy loss through a linear local head, and all layers update simultaneously from the same forward pass. The method applies spectral updates to layer weights and scales steps using input RMS to bound weight-induced drift.

2 Method

  • Each layer has its own linear local head and cross-entropy loss; gradients do not flow between layers, and all layers update simultaneously from one forward pass.The setup uses a depth-L ReLU MLP, with no trunk normalization unless stated.Source: §2 · p. 2
  • The spectral update uses momentum orthogonalization for matrices, while biases and heads use Adam.NewtonSchulz5 approximately maps singular values of normalized momentum to one; bias updates are clipped to ϵ RMS and heads use Adam.Source: §2 · p. 2
  • The drift-contract scaling uses input RMS and a shape factor to set the learning rate so weight-induced pre-activation drift is bounded per step by ϵκl.For square and expanding layers the shape factor is 1; the bound is conditioned on the current input and the EMA tracking batch RMS.Source: §2 · p. 3
  • The bound covers each layer’s parameter-induced drift, not drift propagated from simultaneous upstream updates; bias budgeting brings total per-layer parameter-induced drift to about 2ϵ.The measured exact-scaled variant has a p99 ratio of 1.09 and maximum of 1.19 across layers.Source: §2 · p. 3

3 Experimental setup

Experiments use CIFAR-10 MLP classifiers with local heads, varying width and depth, and assess test accuracy alongside activation drift and feature effective rank. The protocol includes prospectively specified controls and reports compute details.

3 Experimental setup

  • CIFAR-10 experiments use flattened 3072-feature images, widths 128–2048, depths 12–48, batch size 256, and 12 epochs.The setup uses 20,000 training and 5,000 test examples, train-statistic normalization, and He initialization.Source: §3 · p. 3
  • The protocol fixed auxiliary-parameter Adam learning rate at 3e-3 in every arm, detected divergence through activations as well as weights, and prospectively specified both possible outcomes of each control.Sweeps used 2 seeds, while headline configurations used 5.Source: §3 · p. 3
  • Synthetic experiments run in about one hour on a laptop CPU, while CIFAR campaigns use one RTX 4070 Ti with CuPy.CPU and GPU paths agree at 0.949 versus 0.948 on the regression check.Source: §3 · p. 3

4 Results

Across CIFAR-scale MLP results, spectral updates transfer one setting across width and depth, while local Adam needs retuning and still degrades sharply with depth. Controls attribute transfer to spectral scaling; the drift contract adds an input-conditioned per-layer bound, and the stability advantage is conditional on the absence of trunk normalization.

4.1 Width transfer with an interior optimum

  • The drift-budget optimum is the same interior value across widths, whereas Adam’s optimal learning rate shifts from 1e-3 at width 128 to 3e-4 at width 512 and beyond.The contract accuracy curve peaks at ϵ = 0.003 on every width; measured drift at fixed ϵ is width-invariant within 15 percent.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2 Depth: the main result

  • At width 128, Adam loses about 13 accuracy points from depth 12 to 48 despite per-depth retuning, while the contract loses about 3 points; Adam’s width-128-optimal setting scores 19 percent at depth 48.The spectral arms use ϵ = 0.003 at all three depths.Source: §4.2 · p. 4, Table 1 · p. 4

4.3 Five-seed statistics

  • The contract beats local Adam by 1.2 to 2.9 points at widths 128, 512, and 1024 in five-seed comparisons; the width-2048 gap is +1.0 on two seeds.The width-2048 result has larger Adam seed variance, so the authors do not infer a trend from it.Source: §4.3 · p. 4, Table 2 · p. 5

4.4 Stability and the normalization caveat

  • Without trunk normalization, local spectral updates are stable across a 333x learning-rate range; with RMSNorm and weight decay, global Muon has the largest range at 300x.Under RMSNorm and weight decay, the local spectral range narrows to 30x; the authors identify absent normalization as the scope where its stability benefit is largest.Source: §4.4 · p. 5

4.5 Attribution controls

  • The fixed-lr spectral optimum transfers across four widths and both extra depths, supporting spectral scaling—not the contract—as the source of setting transfer.At depth 24 the fixed-lr update matches the contract at 45.6; at depth 48 the contract leads by 2.5 points, 42.7 versus 40.2, on two seeds. The contract adds a small gain over the best fixed lr where measured and a per-layer drift bound.Source: §4.5 · p. 5
  • Bias ablations do not explain the residual drift floor; constant-norm orthogonalized steps do, because they never anneal.The tested bias variants are frozen, normalized momentum, and clipped Adam.Source: §4.5 · p. 6

4.6 Mechanism

  • At matched settings and widths 512 and up, layer-12 feature effective rank is 27 to 47 under spectral updates versus 6 to 23 under Adam, but the causal mechanism remains unsettled.The authors note that random or inverted spectra can match Muon in global training and leave spectrum ablations for future work.Source: §4.6 · p. 6

5 Related work

Prior work studied spectral update geometry and transferable learning rates mainly in global training, while local-learning work established auxiliary-loss and greedy layer-wise approaches. The paper distinguishes its contribution as applying spectral geometry to auxiliary-head local gradients and deriving step sizes from an inter-layer drift bound.

5 Related work

  • Muon uses orthogonalized momentum and spectral-norm-constrained steepest descent, with analyses of operator and modular norms and scaling to LLM training (Bernstein and Newhouse, 2024; Liu et al., 2025).Related spectral work includes vision transformers (Southworth et al., 2026), layer-wise LMO optimization (Riabinin et al., 2025), and a mechanistic challenge (Shumaylov et al., 2026).Source: §5 · p. 6
  • Width-transferable learning rates were studied for global training through muP and modular norms, while Ishikawa et al. (2024) showed width transfer for local predictive-coding and target-propagation losses.The present setting instead uses auxiliary classification heads, extends transfer to depth, and derives its step size from a per-layer drift bound.Source: §5 · p. 6
  • Local learning work established auxiliary local losses, decoupled greedy learning, and forward-forward, but the cited literature had not applied spectral update geometry to auxiliary-head local gradients or derived step sizes from an inter-layer drift bound.The cited local-learning work includes Nøkland and Eidnes (2019), Belilovsky et al. (2019), and Hinton (2022).Source: §5 · p. 6

6 Limitations

The evaluation is limited to CIFAR-scale MLP classifiers with linear local heads and 12 fixed epochs for the headline results.

  • CIFAR-scale MLP classifiers with linear local heads and 12 fixed epochs define the headline evaluation scope.Source: §6 · p. 6
  • Width 2048 and two Adam configurations use two seeds rather than five, outside the seed sweep.Source: §6 · p. 6
  • The drift bound covers weight-induced change at the current input, not propagated or off-distribution drift.Source: §6 · p. 6

7 Conclusion

On these benchmarks, a domain-calibrated drift budget can be reused as models grow or deepen, while the per-layer drift bound remains conditional and two decentralized-training challenges remain.

7 Conclusion

  • Calibrate ϵ once per domain on a small model, then reuse it unchanged as the model grows or deepens on these benchmarks.Source: §7 · p. 7
  • The contract caps each layer’s per-step contribution to behavioral change, but does not cover drift propagated from upstream layers.Source: §7 · p. 7
  • Forward-pass network latency and local-loss quality at language-model scale remain unresolved for decentralized training.Source: §7 · p. 7

Reproducibility

The implementation and experiment materials are available as a compact NumPy-based release with raw results and checkpoints.

Reproducibility

  • The code is pure NumPy, with an optional CuPy backend, in a single library file plus one runner per experiment.Source: Reproducibility · p. 7
  • JSON checkpoints are saved after every run, and code and raw results are available in the project repository.Source: Reproducibility · p. 7
  • Synthetic results reproduce in about one hour on a laptop CPU.Source: Reproducibility · p. 7
Loading 2609.26811v1…