PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov

arXiv:2610.11730v1cs.LGmath.OC

TL;DR

Standard weight decay ignores the spectral structure of weight matrices. The paper introduces a decoupled nuclear-norm update with additive spectral shrinkage, and reports lower effective rank and improved compression and noisy-label classification outcomes.

  • Problem

    Standard weight decay treats each weight matrix as a vector and does not explicitly favor removing small singular values.

    Source: §1 · p. 1

  • Method

    Spectral weight decay replaces the squared Frobenius penalty with a decoupled post-step nuclear-norm correction that shrinks singular values additively.

    Source: §1 · pp. 1–2

  • Results

    Across 124M–500M LLaMA models it lowers effective rank and improves compression and inference speed at matched validation loss; at 60% label noise, mean clean-test accuracy gains reach 17.78 points on MNIST.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Low-effective-rank weights admit efficient truncated-SVD compression, reducing parameter count and multiplication cost when the retained rank is sufficiently small.

    Source: §1 · p. 2

    Preparing an existing checkpoint for low-rank compression without a substantial loss increase remains open.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.

1 Introduction

The paper replaces matrix-agnostic Frobenius weight decay with a decoupled nuclear-norm update that shrinks singular values additively. It connects the update to approximate proximal descent and reports compression and noisy-label gains.

1 Introduction

  • Unlike the described decoupled SLORR-Nuc extension, which uses a pre-step correction, spectral weight decay uses the post-step correction.SLORR-Nuc approximates the nuclear-norm polar factor.Source: §1 · pp. 1–2
  • Because additive shrinkage acts more strongly on smaller singular values, the resulting low-effective-rank weights admit efficient truncated-SVD compression.Source: §1 · p. 2
  • Across 124M–500M LLaMA models, spectral weight decay lowers effective rank and improves compression and inference speed at matched validation loss.At 500M parameters and a 4% distortion budget, it reaches 1.89× compression and 1.18× GPU inference speedup.Source: §1 · p. 2
  • At 60% label noise, mean clean-test accuracy gains over matched ℓ2 regularization reach 17.78 points on MNIST and 4.59 points across four BERT-base tasks.Source: §1 · p. 2
  • Spectral weight decay replaces the squared Frobenius penalty with nuclear-norm regularization and applies additive shrinkage in the updated weights’ singular basis.The update is connected to approximate proximal descent.Source: §1 · pp. 1–2

2 Related Work

Related work covers conventional weight decay, direct nuclear-norm regularization, low-rank training, and post-training compression. Spectral weight decay differs by retaining dense weights during training and applying its nuclear-norm update post-step.

2 Related Work

  • Conventional weight decay is commonly used for generalization and training stability, but its effects depend on the optimizer and training setup.The squared ℓ2 penalties of two factors over all factorizations of W = UV^T are equivalent to a nuclear-norm penalty on W.Source: §2 · p. 2
  • Direct nuclear-norm regularization targets weight matrices without fixing rank, but exact gradients and proximal updates require spectral decompositions.Prior work includes periodic singular-value thresholding and SLORR’s polar-factor approximations; this paper analyzes a post-step nuclear-norm update.Source: §2 · p. 2
  • NuMuon regularizes Muon’s optimizer update direction, rather than the weights, to promote low-rank updates.Source: §2 · p. 3
  • Cuttlefish and TRP impose low-rank structure during training, whereas spectral weight decay keeps dense weights and delays rank selection until compression.Source: §2 · p. 3
  • Post-training compression methods include transformed or activation-informed SVD, learned truncation, and fine-tuning for SVD compression.The paper applies spectral weight decay throughout pretraining and evaluates compression and inference with SVD-LLM.Source: §2 · p. 3

3 Why Low-Rank Structure Matters

The section connects spectral weight decay’s lower effective rank to low-rank compression, then evaluates its compression, inference, and noisy-label performance. Across the reported settings, it improves compression and inference at matched validation loss and yields higher clean-test accuracy under label noise.

3 Why Low-Rank Structure Matters

  • Spectral weight decay replaces standard pre-step ℓ2 decay in Adam with a post-step correction, promoting low-rank structure that is evaluated for compression, inference, and noisy-label robustness.The update is framed as an analogy to ℓ1 regularization in linear regression.Source: §3 · p. 3, §3.1 · p. 3

3.1 Effective rank and why it matters

  • At λ = 0.7, mean effective rank falls 2.17-fold vs λ = 0, with 1.10% higher validation loss; truncating to ceil(erank) preserves quality, while lower ranks degrade sharply.Effective rank decreases as the spectrum becomes more concentrated.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Compression and inference

  • At 500M and 4% distortion, compression is 1.89× vs 1.14× after standard weight decay, while inference speedup is 1.18× vs 1.01×.Factorized models outperform dense counterparts only at large batch sizes; latency is measured at batch size 256.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 Robustness to noisy labels

  • At 60% label noise, mean clean-test accuracy gains over matched ℓ2 regularization reach 17.78 points on MNIST and 4.59 points across four BERT-base tasks.The evaluation uses fixed-horizon training and final checkpoints, without clean-validation early stopping.Source: Figure 4 · p. 5, §3.3 · pp. 5–6

4 From Weight Decay to Spectral Shrinkage

The section derives spectral weight decay as an additive singular-value correction applied after the loss-only optimizer step. It motivates this order through its connection to approximate nuclear-norm proximal descent and analyzes how update order affects the correction.

4 From Weight Decay to Spectral Shrinkage

  • Decoupled weight decay first forms the loss-only candidate Z, then applies regularization to that candidate rather than to the current weights.This post-step choice regularizes the argument of the proximal operator in proximal-gradient descent.Source: §4 · p. 6

4.1 The spectral update

  • The post-step correction uses the updated weights’ singular directions, approximating a nuclear-norm proximal step through the polar factor.The polar factor is a minimum-Frobenius-norm subgradient of the nuclear norm, and the paper approximates it without an SVD using Newton-Schulz iterations.Source: §4 · p. 6, §4.1 · p. 6
  • Near rank deficiency, pre- and post-step spectral updates can differ by as much as the 2ηλ spectral-norm bound, exceeding the corresponding ℓ2 order difference.When both smallest singular values are at least γ > 0 and the optimizer direction is bounded, the discrepancy is O(η^2λ/γ).Source: §4.1 · p. 7
  • Spectral decay subtracts τ from each singular coefficient, while ℓ2 decay scales each by 1 − τ; smaller singular values receive greater relative reduction.The approximate spectral step can cross zero without setting a coefficient exactly to zero, unlike the exact nuclear-norm proximal update.Source: §4.1 · pp. 6–7

5 Limitations and Future Work

The experiments identify limits on preparing existing checkpoints for low-rank compression and show that spectral regularization’s loss–rank trade-off depends on the optimizer. Fine-tuning from a conventional weight-decay checkpoint incurs more loss at matched rank, while Muon and Lion behave differently under spectral regularization.

5.1 Fine-tuning for low-rank compression

  • Fine-tuning a 124M-parameter ℓ2 weight-decay checkpoint incurs substantially more validation loss at the same rank than spectral regularization during pretraining.Fine-tuning also requires a much larger λ before rank changes appreciably.Source: §5.1 · p. 7, Figure 5 · p. 7
  • Preparing an existing checkpoint for low-rank compression without a substantial loss increase remains open.The cited SLORR work also reports difficulty fine-tuning LLMs with low-rank regularization while preserving general capabilities.Source: §5.1 · p. 8

5.2 Spectral regularization with matrix-aware optimizers

  • Lion shows a more favorable loss–rank trade-off than Muon, whose substantial rank reduction comes with a steep validation-loss increase.A mean effective rank of about 300 requires λ = 0.8 for Adam, λ = 0.4 for Muon, and λ = 4 for Lion, so the regularization scale needs optimizer-specific calibration.Source: §5.2 · p. 8, Figure 6 · p. 8
  • Muon and SOAP produce higher effective ranks than the other compared optimizers under a shared ℓ2 weight-decay coefficient of 0.1.The authors suggest these dynamics may oppose the spectral concentration encouraged by spectral weight decay.Source: §5.2 · p. 8, Figure 6 · p. 8

6 Conclusion

The paper concludes that post-step nuclear-norm regularization produces additive spectral shrinkage and low-effective-rank weights. Across its evaluated LLaMA models and noisy-label classification settings, it reports compression and clean-test accuracy benefits, while noting that outcomes depend on when and how the regularization is applied.

6 Conclusion

  • The post-step nuclear-norm correction shrinks singular values additively, with a stronger relative effect on smaller values, and is connected to approximate proximal descent.The paper reports that pre- and post-step spectral updates can differ more substantially than their ℓ2 counterparts near rank deficiency.Source: §6 · p. 9
  • At 500M parameters and a 4% distortion budget, spectral weight decay achieves 1.89× compression and 1.18× GPU inference speedup.Across 124M to 500M LLaMA models, the resulting low-effective-rank structure improves post-training compression under matched validation-loss budgets.Source: §6 · p. 9
  • Under fixed-horizon noisy-label training, spectral regularization improves clean-test accuracy on image and text classification tasks.The paper describes these observations as consistent with concentrating the weight spectrum limiting capacity to fit corrupted labels.Source: §6 · p. 9
  • The benefits depend on application: short fine-tuning from a conventional weight-decay checkpoint incurs a larger loss penalty, and Muon runs show a steeper loss increase as rank decreases.The authors identify adapting spectral regularization to matrix-aware optimizers and reducing checkpoint-preparation costs as open problems.Source: §6 · p. 9

A.1 Proof of the spectral update-order bound

The pre/post-step spectral updates can differ substantially near rank deficiency, even when the task step is small. The post-step rule also approximates a nuclear-norm proximal step, with discrepancy limited to the sub-threshold singular-value tail.

A.1 Proof of the spectral update-order bound

  • The pre/post-step difference is bounded by 2η²λ||Δ||F divided by σmin(W) + σmin(Z), so the bound grows as the smallest singular values shrink.The bound applies to the rectangular-matrix polar factors without requiring gaps between nonzero singular values.Source: §A.1 · p. 13
  • Near rank deficiency, a vanishing task step can flip a diagonal entry’s sign and leave the two polar factors a fixed distance apart.The example has smallest singular values η/2 and η(1 + η)/2, while ||Dη||F ≤ 3/2.Source: §A.1 · p. 13

A.2 Proximal interpretation

  • Exact thresholding and Newton–Schulz reach similar effective rank, but exact thresholding ends with higher validation loss.Applying the exact update only once every 10–50 steps leaves the comparison unchanged.Source: Figure 7 · p. 14, §A.2 · p. 15
  • For singular values σi ≥ τ, post-step spectral decay matches the nuclear-norm proximal correction; below τ, exact thresholding instead zeroes the direction.Thus, with an exact polar factor, the approximation discrepancy is confined to the sub-threshold tail.Source: §A.2 · p. 14

A.3 Empirical comparison of pre- and post-step spectral weight decay

Controlled recovery tasks generally favor post-step spectral weight decay, while language-model pretraining shows sensitivity to update order at higher learning rates. At small learning rates, the LLM trajectories differ little, so the authors adopt post-step as the default.

A.3 Empirical comparison of pre- and post-step spectral weight decay

  • Post-step decay reduces mean reconstruction error by 1.4% to 5.0% in matrix denoising; its pre/post ℓ2 difference stays below 0.02%.With NS5, post-step also performs better on average, though gains fluctuate more across noise levels.Source: Figure 8 · p. 15, §A.3 · p. 15
  • In matrix sensing, post-step NS5 gains range from 1.2% to 6.1%, while the ℓ2 baseline again differs by only tiny fractions of a percent.Post-step has lower mean reconstruction error at every displayed noise level with both the exact polar factor and NS5.Source: §A.3 · p. 15, Figure 9 · p. 16
  • At learning rate 0.010, post-step reaches lower effective rank but higher final validation loss; at 0.005, the trajectories nearly coincide.The accompanying rank change prevents a clear preference between update orders in this sweep.Source: Figure 10 · p. 16, §A.3 · p. 16
  • Post-step spectral decay performs better on average in the synthetic recovery tasks, with generally modest gains.The comparison uses separately tuned pre- and post-step ℓ2 weight decay as a baseline.Source: §A.3 · pp. 15–16

B Experimental Details and Additional Results

The additional experiments use LLaMA pretraining on FineWeb-Edu and compare spectral weight decay with Prehab and Cuttlefish. Spectral weight decay lowers effective rank in both matched comparisons, although the Prehab comparison incurs a validation-loss cost too high for practical compression.

B.1 Language-model pretraining setup

  • LLaMA pretraining uses FineWeb-Edu, 1,024-token sequences, an effective batch of 128 sequences, and five Newton–Schulz iterations for spectral decay.The listed configurations summarize the tested architectures and reference pretraining runs.Source: §B.1 · p. 17, Table 1 · p. 17

B.2 Comparison with Prehab

  • At comparable validation loss, spectral weight decay reduces effective rank more than Prehab, but its 2.46% relative validation-loss increase after 1,000 steps is too costly for practical compression.After 1,000 steps, mean effective rank is reduced by 15.9%.Source: Figure 11 · p. 17, §B.2 · p. 17

B.3 Comparison with Cuttlefish

  • Across all four matched settings, spectral weight decay reaches substantially lower final effective rank than Cuttlefish at comparable validation loss.Cuttlefish waits for stable layer ranks, factorizes layers, then continues training the low-rank factors.Source: §B.3 · pp. 17–18, Figure 12 · p. 18

B.4 Compression method and procedure

The compression experiments select SVD-LLM based on its validation-loss performance among compressed models. The procedure leaves embeddings and the output head uncompressed and chooses truncation ranks per matrix, trading parameter savings against validation loss.

B.4 Compression method and procedure

  • SVD-LLM has the lowest validation loss among the compressed models and is selected for subsequent comparisons.The 124M spectral-weight-decay checkpoint is evaluated with five singular-value-based compression methods.Source: Table 2 · p. 18, §B.4 · p. 18
  • Token embeddings and the output head remain uncompressed because truncating them sharply degrades model quality; each remaining matrix gets its own truncation rank.Effective rank varies substantially across transformer blocks.Source: §B.4 · p. 18
  • A rank-r factorization stores r(m + n) parameters instead of mn, saving memory at the cost of higher validation loss.Source: §B.4 · p. 19

B.5 Noisy-label robustness results

Tables 3–5 report final-checkpoint clean-test accuracy across MNIST MLP, MNIST GRU, and BERT-base evaluations. The supplied passages describe the evaluation and model setups but do not provide the accuracy values.

B.5 Noisy-label robustness results

  • Tables 3–5 report final-checkpoint clean-test accuracy as mean ± one sample standard deviation over five label-corruption seeds.The passages do not include the table cell values, so they do not establish which regularizer performs better.Source: §B.5 · p. 19, Table 3 · p. 19, Table 4 · p. 19
  • Model initialization, training order, data splits, and regularization coefficients are fixed; standard deviations reflect corruption-seed variability only.Source: §B.5 · p. 19
  • The MNIST evaluations use a four-layer, width-2,048 MLP or a two-layer GRU with hidden dimension 1,280.The MLP flattens each 28 × 28 image; the GRU reads 28 vectors of dimension 28.Source: §B.5 · p. 19
  • BERT-base has approximately 110M parameters, with embeddings and blocks 1–8 frozen and blocks 9–12, the pooler, and classifier trained.Inputs are truncated to 128 tokens, and checkpoints are taken at a fixed training horizon.Source: §B.5 · p. 19
Loading 2610.11730v1…