PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Architecture, Different Capacity: Optimizer-Induced Spectral Scaling Laws

Nandan Kumar Jha, Brandon Reagen

arXiv:2605.21803v1cs.LG

TL;DR

The paper asks whether optimizers that yield similar loss scaling also produce similar internal representation scaling. It measures FFN eigenspectra across effective-rank probes, token-frequency regimes, and architectural interventions, finding that optimizer choice substantially changes spectral capacity and geometry even under fixed architecture and matched loss.

  • Problem

    Existing scaling laws treat the optimizer largely as fixed, leaving open whether similar loss scaling implies similar internal representation geometry.

  • Method

    The study compares optimizers by measuring FFN eigenspectra with soft and hard effective ranks across widths, token-frequency regimes, and architectural interventions.

  • Results

    The same architecture exhibits substantially different optimizer-dependent spectral scaling laws, and matched perplexity does not imply matched representation geometry.

  • Takeaways & Limitations

    Optimization is a first-class axis of representation scaling and should be considered jointly with architecture when analyzing effective capacity.

  • Takeaways & Limitations

    Evidence is limited to GPT-style decoder-only models at 160M and 350M parameters, dense FFNs, selected optimizers, and a fixed data recipe; 1B+ testing remains open.

Abstract

from arXiv · show

Scaling laws have made language-model performance predictable from model size, data, and compute, but they typically treat the optimizer as a fixed training detail. We show that this assumption misses a fundamental axis of representation scaling: how effectively the optimizer converts added FFN width into utilized spectral capacity. Using eigenspectra of feed-forward network representations, measured through soft and hard spectral-ranks, we find that \emph{the same Transformer architecture realizes markedly different spectral scaling laws when trained with different optimizers}. Holding architecture and width schedule fixed, AdamW exhibits weak hard-rank scaling ($β$=0.44) on rare-token (TAIL) representations where learning is known to be hardest, whereas Muon achieves linear scaling ($β$=1.02) in the same regimes, a $2.3\times$ increase in the scaling exponent. This difference is not reducible to validation loss: AdamW configurations can match low-rank Dion variants in perplexity, under extended training, while exhibiting sharply different spectral geometry, demonstrating that matched loss does not imply matched representation structure. Hard--soft rank asymmetry further reveals that optimizers differ not only in how much capacity is realized, but also in how that capacity is structured across eigenmodes. To disentangle optimizer effects from architectural ones, we compare against architectural interventions (e.g., attention rank and positional encoding), and find that optimizer-induced spectral shifts often exceed the architectural effects. These results suggest optimization as a first-class axis of representation scaling, motivating optimizer--architecture co-design.

1 Introduction

The paper shows that optimizer choice is a fundamental axis of representation scaling: fixed Transformer architectures can realize different FFN spectral-capacity laws and geometries. These differences persist despite matched loss and interact with token frequency and architectural interventions.

  • Optimizer-induced spectral scaling: Optimizer choice changes FFN spectral scaling laws even when architecture, width schedule, and training data are fixed.The study frames spectral scaling as a property of the architecture–optimizer pair rather than architecture alone.
  • Optimizer-induced spectral scaling: βhard = 0.29 for AdamW versus βhard = 0.82 for Muon, while soft-rank exponents remain within βsoft ∈[0.66, 1.01].Hard rank is more sensitive to concentration in dominant eigenmodes, whereas soft rank captures entropy-weighted spectral spread.
  • Spectral geometry: ∆1,2 = 0.37 for AdamW versus ∆1,2 ≈0.14 for Muon and Dion(1/2), showing optimizer-dependent hard–soft asymmetry.The gap indicates whether added width expands diffuse low-variance directions more than dominant representational directions.
  • Loss versus geometry: Matched perplexity does not imply matched representation geometry: AdamW can match matrix-aware variants under extended training while spectra remain structurally distinct.The divergence is not explained by learning-rate tuning, convergence speed, or final validation loss.
  • Frequency-dependent capacity: Optimizer-dependent capacity allocation varies across token frequencies, with especially weak AdamW hard-capacity scaling for rare tokens and the largest AdamW-to-Muon gain in MID.The results connect optimizer geometry to how representation capacity is distributed across the token-frequency spectrum.
  • Optimizer–architecture interaction: Optimizer-induced spectral shifts often dominate or reshape interventions in attention rank and positional encoding.The comparisons include smaller spectral changes from increased per-head attention rank and optimizer-dependent redistribution after RoPE removal.

2 Related Work

Prior work established resource-based loss scaling, spectral capacity measures, and optimizer geometry as separate concerns. This paper connects them by studying how optimizer geometry interacts with FFN representations and architectural priors.

  • Scaling laws and optimizer awareness: Classical scaling laws relate validation loss to model size, data, and compute, while treating the optimizer as a fixed training choice.Related work also reports architecture-dependent scaling exponents in other model domains.
  • Spectral capacity: Prior spectral-scaling work measured FFN latent-space utilization under a fixed optimizer, leaving optimizer-dependent representation scaling open.Effective-rank research provides tools for characterizing the dimensionality and utilization of learned representations.
  • Optimizer geometry: Muon orthogonalizes matrix updates, while Dion offers a rank-controlled intervention for analyzing optimizer-induced geometry.Matrix-based optimizers can be competitive in LLM pretraining, with outcomes depending on tuning, scale, and evaluation protocol.
  • Architecture–optimizer interaction: Earlier work shows that architectural priors can be transferred into optimizers and that optimizer geometry modulates variance redistribution within fixed-width FFNs.These results motivate analyzing architecture and optimization jointly rather than as independent design choices.

3 Methodology

The methodology measures FFN spectral capacity through complementary effective-rank probes, probe locations, and token-frequency strata. Scaling exponents quantify how efficiently added FFN width becomes latent capacity at different concentration sensitivities.

  • Probe locations: Pre-activation states expose optimizer-shaped linear geometry, while post-activation states measure capacity realized after the FFN nonlinearity.Comparing both probes separates expansion geometry, nonlinear redistribution of spectral mass, and capacity passed onward.
  • Spectral measurement: Trace-normalized eigenspectra make variance allocation comparable across probe points, layers, and widths.The normalized spectrum represents how variance is distributed across FFN latent directions.
  • Rényi effective rank: Rényi effective ranks provide a continuum of spectral-capacity measures with different sensitivities to weak and dominant eigendirections.Lower orders emphasize diffuse spectral support, while higher orders emphasize dominant modes.
  • Rényi effective rank: R1 is the soft rank for diffuse spectral spread, whereas R2 is the harder, concentration-sensitive participation ratio.The full sweep uses α ∈{0.5, 1, 1.5, 2, 3, 5}, with α = 1 and α = 2 as the main anchors.
  • Scaling-law fits: ∆1,2 = βsoft −βhard measures exponent-level hard–soft asymmetry, with larger values indicating greater expansion of low-variance directions relative to dominant ones.The exponents are fitted from R(D) ∝Dβ as FFN width D varies.
  • Frequency stratification: Token-frequency stratification divides token types into HEAD, MID, and TAIL regimes before computing covariance spectra and rank metrics.HEAD covers the most frequent third of occurrence mass, MID the next third, and TAIL the remaining lower-frequency types.

4 Optimizer-Induced Spectral Scaling Laws

With architecture and width schedule fixed, optimizer choice substantially changes how FFN width becomes soft and hard spectral capacity. These effects vary across token frequencies, persist under extended training and larger scale, and often exceed tested architectural shifts.

  • 4.1 Spectral Scaling Laws Are Optimizer-Dependent: βhard(TAIL) is 0.44 for AdamW versus 1.02 for Muon and 1.04 for NorMuon, while soft-rank exponents remain comparatively clustered.TAIL and MID are the most diagnostic regimes for optimizer-induced scaling effects.
  • 4.1 Spectral Scaling Laws Are Optimizer-Dependent: ∆βMID = +0.69, ∆βTAIL = +0.58, and ∆βHEAD = +0.33 for AdamW-to-Muon hard-rank scaling gains.The MID gain is roughly 2.1× the HEAD gain.
  • 4.1 Spectral Scaling Laws Are Optimizer-Dependent: AdamW maintains positive hard–soft asymmetry, whereas Muon and NorMuon nearly eliminate it for MID and TAIL tokens.This indicates that optimizers differ in whether added width becomes diffuse spectral spread or dominant-mode capacity.
  • 4.2 Matched Loss Does Not Imply Matched Spectral Geometry: βhard drops from 0.29 at 6K steps to 0.03 at 12K steps for AdamW, despite AdamW 12K matching Dion (1/16) in perplexity.The degradation appears across HEAD, MID, and TAIL regimes, so longer AdamW training does not restore dominant-mode scaling.
  • 4.3 Update Rank Constrains Hard-Capacity Scaling: βhard falls from 0.88 at r = 1/2 to 0.40 at r = 1/16 for Dion TAIL representations, while βsoft declines more gradually from 0.95 to 0.72.The corresponding TAIL asymmetry rises from +0.07 to +0.31, making low update rank a bottleneck for hard capacity.
  • 4.5 Optimizer-Induced Scaling Effects Dominate the Attention-Rank Interventions: At 350M, Muon reaches βhard = 1.13 while AdamW reaches 0.39, and optimizer-induced gains exceed attention-rank shifts in 28 of 30 comparisons.The only attention-rank exceptions occur in HEAD hard-rank scaling; the optimizer–architecture match also changes with the intervention.

5 Discussion and Conclusion

The paper argues that spectral scaling depends on the architecture–optimizer pair: optimizer geometry determines how added FFN width is expressed, while architectural effects are not optimizer-independent. The study is limited to controlled dense-FFN experiments and spectral measures that do not fully characterize downstream behavior.

  • Discussion and Conclusion: Optimizer geometry determines whether added FFN width becomes diffuse or dominant-mode spectral capacity.Muon-style optimizers convert width more efficiently, whereas AdamW contributes primarily to diffuse spectral capacity.
  • Discussion and Conclusion: Update rank controls how efficiently added FFN width becomes usable hard spectral capacity, while orthonormalization alone is insufficient under aggressive rank constraints.Low-rank Dion approaches AdamW-like scaling.
  • Discussion and Conclusion: Optimizer-induced representation geometry persists beyond convergence differences, so matched validation loss can coexist with distinct spectral scaling behavior.The paper treats this as evidence that the geometry is not a transient training artifact.
  • Discussion and Conclusion: Optimizer-induced gains exceed tested attention-rank effects in some regimes, while architectural interventions can redistribute optimizer differences or change which optimizer best matches an architecture.The broader conclusion is that effective representational capacity is realized by an architecture–optimizer pair.
  • Limitations: The evidence is limited to dense FFN architectures, four optimizer families, a fixed data recipe, and models at 160M and 350M parameters.The authors identify 1B+ experiments and broader architectures, protocols, and optimizers as needed for stronger generality.
  • Limitations: Soft and hard spectral ranks quantify FFN capacity conversion but do not fully characterize downstream task behavior.The paper highlights causal tests of update geometry and spectral concentration as future work.

A.2 Training protocol

The experiments use controlled Transformer training runs with fixed protocols, FFN-width sweeps, optimizer-specific updates, and frequency-stratified spectral measurements. Validation results provide loss-level context, showing that Muon and NorMuon generally perform best while extended AdamW can match low-rank Dion in perplexity without eliminating spectral differences.

  • Training protocol: Each optimizer–width combination is trained independently with a constant learning rate, 20% linear cooldown, no warmup, BF16 precision, and no gradient clipping.Extended-training controls run AdamW for 12,000 iterations with other settings fixed.
  • Training protocol: Muon, NorMuon, and Dion use the matrix-aware update recipe with Lion for non-matrix parameters and spectral-condition learning-rate scaling.AdamW uses separate beta parameters, while matrix-aware optimizer hyperparameters are reported separately.
  • Token-frequency stratification: HEAD, MID, and TAIL regimes are fixed across optimizers and widths using cumulative token-occurrence mass from 10.26B FineWeb-Edu tokens.The regimes each contain approximately one third of occurrences but sharply different numbers of token types.
  • Spectral measurements: FFN representations are aggregated over held-out validation tokens, converted to normalized covariance eigenspectra, and summarized with layer-averaged spectral metrics.Frequency-stratified metrics are computed separately within each token-frequency regime.
  • Validation context: Muon and NorMuon consistently achieve the strongest validation perplexity, while extended AdamW reaches perplexity comparable to Dion (r = 1/16) but remains worse than Muon and NorMuon.The comparable loss does not remove the spectral-geometry differences reported elsewhere.

C Rényi Effective Rank Analysis: Where Optimizer-Induced Capacity Forms

Rényi-order analysis localizes optimizer-induced spectral differences before and after the FFN nonlinearity, showing that activation can reorder effective capacity across optimizers.

  • Rényi-rank framework: R1 and R2 anchor a Rényi effective-rank continuum that probes diffuse versus dominant-mode spectral capacity.Lower orders emphasize weak eigendirections, whereas higher orders increasingly emphasize dominant modes.
  • Pre-activation geometry: Across Rényi orders, pre-activation rank follows NorMuon > Muon > Dion(r = 1/2) > Dion(r = 1/16) > AdamW.At α = 1, rank rises from 126.0 for AdamW to 411.3 for NorMuon.
  • Post-activation geometry: At α = 2, the FFN nonlinearity reorders post-activation ranks, with Dion (r = 1/2) highest and AdamW lowest.Post-activation rank therefore captures effective latent capacity differently from the optimizer-shaped precursor geometry.
  • Nonlinear reinjection: At α = 2, AdamW’s reinjection ratio is 51.9, versus 4.8 for Muon and 4.3 for NorMuon, yet AdamW has the lowest post-activation R2.The comparison indicates that larger nonlinear reinjection does not correspond to better spectral capacity.
  • Rényi-order reversals: At α = 5, Dion (r = 1/16) has lower pre-activation rank than Muon and NorMuon but slightly higher post-activation rank than both.The reversal is 31.8 versus 68.3 and 71.4 before activation, compared with 221.7 versus 217.3 and 198.7 after activation.

D Layer-Wise Robustness of Optimizer-Induced Spectral Scaling

Layer-wise diagnostics show that aggregate optimizer-dependent scaling patterns extend across depth rather than arising from only a few layers, although hard-rank scaling is more heterogeneous.

  • Layer-wise methodology: Layer-wise exponents βℓ are fit independently across FFN widths and summarized by medians, IQRs, and positive-exponent fractions.These diagnostics test whether aggregate scaling trends are broadly reflected across network depth.
  • Soft-rank robustness: Soft-rank exponents are positive for nearly all layers, with large medians for Muon-style optimizers across regimes.NorMuon reaches a median soft-rank exponent of 0.831 in HEAD and 0.9 in the reported continuation.
  • Hard-rank heterogeneity: AdamW’s median hard-rank exponents remain below 0.40 in every regime, with 75% of layers scaling positively in HEAD and MID.Hard-rank variation across depth is stronger than for soft rank, reflecting its more selective focus on dominant modes.
  • Hard-rank separation: In TAIL, median hard-rank exponents are 0.746 for Muon, 0.714 for NorMuon, and 0.751 for Dion (1/2).These values preserve the aggregate separation between orthonormal-update optimizers and AdamW at layer level.
  • Low-rank Dion: Dion (1/16) has positive hard-rank scaling in every layer but remains capacity-limited, with TAIL median βℓ = 0.404.This is lower than 0.746 for Muon, 0.714 for NorMuon, and 0.751 for Dion (1/2).
  • Depth profiles: Depth profiles vary especially for hard rank, yet the optimizer ordering remains visible across many layers.AdamW generally shows weaker hard-rank scaling, while Muon, NorMuon, and Dion (1/2) often show larger positive exponents.

E Hard-Rank Dynamics Under Extended AdamW Training

Extended AdamW training disrupts the width–capacity relationship for post-activation hard rank, especially for wider FFNs and rare-token representations.

  • Scaling-law breakdown: Extended AdamW training breaks the power-law relationship between FFN width and hard rank even while validation perplexity improves.The analysis focuses on hard rank because extended training produces diminishing βhard(TAIL).
  • TAIL dynamics: In TAIL, the 8d model rises from 278 at 2K to 394 at 4K, then falls to 228 by 8K.The wider model consequently loses its expected dominant-mode capacity advantage.
  • HEAD and MID dynamics: In HEAD, the 8× trajectory declines from roughly 117 near its early peak to about 47 by 12K, while smaller widths converge.The erosion is milder than in TAIL but still weakens width ordering.
  • HEAD and MID dynamics: In MID, the 8× trajectory falls below the 4× trajectory by the end of training.Across regimes, wider configurations are preferentially eroded rather than shifted uniformly.

F The AdamW–Muon Spectral-Scaling Gap Persists Across Learning Rates

Learning-rate sweeps do not remove the AdamW–Muon spectral-scaling gap, supporting optimizer-dependent geometry rather than a simple scalar learning-rate explanation.

  • Hard-rank comparison: In TAIL hard rank, AdamW’s largest reliable exponent is β = 0.44, while Muon’s lowest is β = 0.80 across tested learning rates.Muon’s weakest setting exceeds AdamW’s strongest by 0.36, and the tested exponent ranges do not overlap.
  • Hard-rank comparison: AdamW’s TAIL hard-rank exponent varies from 0.32 to 0.44, whereas Muon’s varies from 0.80 to 1.02.The within-optimizer ranges are smaller than the cross-optimizer separation.
  • Regime dependence: AdamW learning-rate changes redistribute weak hard-rank scaling across frequency regimes rather than producing Muon-like behavior.At 10−3, HEAD improves to βhard = 0.43 while MID collapses to β = 0.00; at 6 × 10−3, TAIL fitting becomes unreliable.
  • Soft-rank comparison: For TAIL soft rank, AdamW’s largest exponent is β = 0.70, versus β = 0.91 for Muon, with the ordering unchanged across tested rates.Muon’s TAIL soft-rank fits also remain stable with R2 ≥0.94.
  • Interpretation: The combined learning-rate and extended-training controls rule out insufficient training time or scalar learning-rate mistuning as simple explanations for AdamW’s weaker hard-rank scaling.The supported interpretation is that optimizer geometry changes how added FFN width becomes usable capacity.
  • Update-rank control: Reducing Dion’s rank fraction from r = 1/2 to r = 1/16 lowers βhard(TAIL) from 0.88 to 0.40, while βsoft falls from 0.95 to 0.72.The intervention primarily reduces hard-rank scaling, especially in MID and TAIL.

H Attention-Rank Effects and Optimizer–Architecture Interactions

Attention-rank interventions interact with optimizer geometry rather than acting as independent shifts. Across nearly all regimes, optimizer changes produce larger spectral-scaling effects and can change which optimizer is best matched to the architecture.

  • Direct effects: 28 of 30 regime–optimizer comparisons show optimizer-induced gains exceeding the absolute attention-rank architectural shift.The exceptions are HEAD hard-rank scaling under AdamW and Muon.
  • Interaction effects: The attention-rank intervention changes the optimizer gap in five of six regimes, with the largest increase in HEAD hard-rank scaling at +0.306.The only negative interaction is MID hard rank, where the best gain decreases by 0.153 but remains large.
  • Optimizer ordering: Under 12 heads, NorMuon has the largest scaling exponent in five of six regimes, whereas under 6 heads, Muon leads in all six.Reducing the number of heads therefore changes optimizer ordering rather than applying a uniform offset.
  • Interpretation: Attention-rank changes and optimizer geometry act as coupled, rather than separable, axes of representation scaling.The comparison separates optimizer-induced gains, absolute architectural shifts, and intervention-induced changes in optimizer gain.

I Optimizer Geometry Expands the Trainable Normalization Space

Optimizer choice changes not only spectral capacity within a fixed architecture but also which normalization-placement architectures can be trained at useful perplexity. Muon-family optimizers remain effective in more difficult partial-PostLN settings, while full PostLN remains unstable.

  • Trainability frontier: At lr=3 × 10^-4, AdamW trains PostLN-25 but diverges for PostLN-50 and PostLN-75.At lr=10^-4, AdamW avoids divergence through PostLN-75 but reaches PPL = 106.7, versus PPL = 40.9 for Muon and PPL = 32.8 for NorMuon.
  • Optimizer differences: NorMuon reaches PPL = 32.8 at PostLN-75, improving over Muon’s PPL = 40.9 in the most aggressive partial-PostLN regime.Muon and NorMuon both achieve PPL = 28.7 at PostLN-25 and nearly identical perplexity at PostLN-50.
  • Scope boundary: Full PostLN remains unstable for all optimizers, despite Muon-family stability in higher partial-PostLN fractions.Dion(1/2) and Dion(1/16) both train PostLN-50 but fail at PostLN-75.
  • Positional processing: Under NoPE, Muon and NorMuon show roughly 3× larger TAIL positional processing at 1d than under RoPE.Muon increases from 0.0380 to 0.1252 and NorMuon from 0.0419 to 0.1313.
  • Depth localization: NoPE shifts Muon’s peak TAIL positional processing to the earliest layers, while NorMuon shifts earlier but remains more distributed.AdamW remains concentrated in mid-to-deeper layers under NoPE.

J.4 NoPE Reverses the Frequency Bias of Positional Processing

Removing explicit positional encoding reverses the frequency bias of FFN-induced positional processing for Muon-family optimizers but not AdamW. This redistribution is optimizer-dependent and affects both token frequency and processing depth.

  • Frequency bias: Under RoPE, all optimizers show a HEAD-favored positional-processing bias, whereas under NoPE, Muon and NorMuon become TAIL-biased.Positive values indicate stronger positional amplification for TAIL tokens.
  • Optimizer dependence: Under NoPE, Muon and NorMuon reorganize toward TAIL-favored positional processing, while AdamW does not show the same reversal.This provides a frequency-level signature of optimizer-dependent positional reorganization.
  • Spectral consequences: Under NoPE, Muon increases HEAD hard-rank scaling while decreasing MID and TAIL, whereas NorMuon decreases hard-rank scaling across all token regimes.The associated changes include stronger TAIL position-dependence and earlier-layer processing for Muon-family optimizers.
Loading 2605.21803v1…