Source-linked AI summary
Spectral Scaling Laws in Language Models: How Effectively Do Feed-Forward Networks Use Their Latent Space?
Nandan Kumar Jha, Brandon Reagen
TL;DR
Existing scaling laws relate model performance to factors such as width, depth, and data, but offer little insight into how components use latent space. The paper audits FFN covariance spectra with a suite of utilization metrics and finds that soft rank scales nearly linearly with width while hard rank grows weakly. The results frame width selection as a trade-off in spectral utilization, within the study's stated model and scale limits.
Problem
Existing empirical scaling laws offer little insight into how architectural components exploit latent space, leaving effective FFN capacity usage and its measurement unresolved.
Source: §1 · p. 1Method
The paper audits post-activation covariance spectra across GPT-2, LLaMA, and nGPT using Hard Rank, Soft Rank, Spectral Concentration, and SUI.
Source: §1 · p. 2Results
Across widths and model sizes from 70M to 250M parameters, soft spectral rank scales near-linearly with FFN width while hard rank plateaus early with weak, noisy scaling.
Source: §1 · p. 2Takeaways & Limitations
The findings identify spectral utilization as an efficiency axis and motivate width-efficient designs through layer-wise scheduling and pruning.
Source: §6 · p. 11The study covers English decoder-only models up to 250M parameters, does not validate multilingual or encoder-decoder settings, and does not establish causality between spectral metrics and perplexity.
Source: Limitations · p. 11
Abstract
from arXiv · showhide
As large language models (LLMs) scale, the question is not only how large they become, but how much of their capacity is effectively utilized. Existing scaling laws relate model size to loss, yet overlook how components exploit their latent space. We study feed-forward networks (FFNs) and recast width selection as a spectral utilization problem. Using a lightweight diagnostic suite -- Hard Rank (participation ratio), Soft Rank (Shannon rank), Spectral Concentration, and the composite Spectral Utilization Index (SUI) -- we quantify how many latent directions are meaningfully activated across LLaMA, GPT-2, and nGPT families. Our key finding is an asymmetric spectral scaling law: soft rank follows an almost perfect power law with FFN width, while hard rank grows only sublinearly and with high variance. This asymmetry suggests that widening FFNs mostly adds low-energy tail directions, while dominant-mode subspaces saturate early. Moreover, at larger widths, variance further collapses into a narrow subspace, leaving much of the latent space under-utilized. These results recast FFN width selection as a principled trade-off between tail capacity and dominant-mode capacity, offering concrete guidance for inference-efficient LLM design.
1 Introduction
The paper frames FFN width selection as a spectral-utilization problem because existing scaling laws offer little insight into how components use latent space. Its central result is asymmetric rank growth: soft rank scales nearly linearly with width, while hard rank grows weakly, consistent with widening adding low-energy directions before expanding dominant modes.
1 Introduction
- The study asks how FFN width affects effective capacity, how many latent directions are used, and how to quantify efficiency beyond FLOPs and loss.Existing scaling laws relate performance to factors such as width, depth, and data size but provide little insight into component-level latent-space use.Source: §1 · p. 1
- A layer-wise spectral audit measures post-activation covariance spectra across GPT-2, LLaMA, and nGPT using four utilization metrics.The metrics are Hard Rank, Soft Rank, Spectral Concentration, and the composite Spectral Utilization Index (SUI).Source: §1 · p. 2
- Soft spectral rank scales near-linearly with FFN width, while hard rank plateaus early with weak, noisy scaling.The paper reports this pattern across widths and model sizes from 70M to 250M parameters.Source: §1 · p. 2
- Widening FFNs predominantly adds low-energy directions while the high-energy mode saturates early.The paper relates this tail-first pattern to diminishing returns from naive width scaling.Source: §1 · p. 2
- Across model families, rank asymmetry persists, practical widths exceed consistently optimal widths, and LayerNorm placement modulates utilization.Post-LN suppresses tail-capacity scaling, whereas Mix-LN improves dominant-mode scaling while preserving near-linear tail growth.Source: §1 · p. 2
2 Related Work
Prior scaling research examines efficiency across compute, deployment, vocabulary, precision, architecture, and domain. This paper positions spectral utilization as a complementary axis focused on how FFN width is used in latent space.
2 Related Work
- Earlier scaling work links loss to compute and identifies compute-suboptimal models, while deployment studies find heavy traffic shifts the compute-optimal point toward smaller models trained on more tokens.Other work maps regimes where capacity, optimizer noise, or embedding quality dominate under fixed budgets.Source: §2 · p. 2
- Efficiency studies also examine vocabulary scaling, numerical precision, and estimation from small pilot runs; spectral-utilization laws target latent-space usage beyond FLOPs.The cited work maps tradeoffs across compute, traffic, vocabulary, and precision.Source: §2 · p. 2
- Related work reports cross-architecture scaling trajectories, a factual-knowledge ceiling, and links between heavy-tailed eigenspectra and implicit self-regularization.Other analyses find small singular values can encode critical information and associate spectral collapse with oversmoothing in attention stacks.Source: §2 · pp. 2–3
- Reported scaling regularities vary with architecture and domain, including architecture-specific scaling exponents and laws for associative-memory matrices and time-series tasks.Other cited work addresses robustness and analytically solvable attention mechanisms.Source: §2 · p. 3
3 Method
The method extracts layer-wise covariance spectra from FFN post-activations and uses them to characterize latent-space utilization. Its metrics capture dominant directions, long-tail usage, variance concentration, and a composite utilization measure.
3.1 Preliminaries and Eigendecomposition
- The FFN notation defines hidden width D and width multiplier α = D/d, and distinguishes pre-activation from post-activation representations.The post-activation is the representation before the down-projection.Source: §3.1 · p. 3
- For each layer and training step, the method samples post-activation token representations, computes their unbiased covariance, and eigendecomposes it for subsequent spectral metrics.The covariance uses all sampled batch tokens without subsampling or statistical approximations.Source: §3.1 · p. 3
3.2 Spectral Rank Metrics
- Hard Rank uses participation ratio to emphasize dominant directions, whereas Soft Rank uses Shannon entropy to capture dimensionality across the full eigenvalue distribution.Hard rank is particularly sensitive to prominent eigenvalues; soft rank captures long-tail variance patterns.Source: §3.2 · pp. 3–4
- SUI harmonically combines hard and soft ranks, while effective dimension translates utilization into an approximate count of active principal components.The harmonic mean penalizes imbalance between the two ranks.Source: §3.2 · p. 4
- Spectral Concentration measures front-loading of variance, distinguishing spectra that use different fractions of the available latent space.Higher concentration indicates variance is concentrated in leading principal components; lower concentration indicates a more uniform distribution.Source: §3.2 · p. 4
4 Experimental Results
Across LLaMA width sweeps, the results show how soft and hard spectral ranks, normalized utilization, effective dimension, and spectral concentration change with FFN width. Soft rank grows near-linearly while hard rank grows sublinearly and noisily, supporting a tail-first account of added capacity.
4.1 Asymmetric Spectral Scaling Laws
- Soft-rank β is 0.88 (70M), 1.07 (130M), and 0.87 (250M), versus hard-rank β of 0.44, 0.60, and 0.41, with noisier hard-rank fits.Figure 2 shows the width sweeps and power-law fits; prose reports that the disparity indicates widening inflates soft rank more than the dominant subspace.Source: Figure 2 · p. 5, §4.1 · p. 5
- As width increases, capacity is allocated tail-first: low-energy directions grow before the dominant high-energy core expands substantially.The authors suggest tail-aware pruning and MoE designs that allocate experts to tail capacity rather than uniformly widening a dense FFN.Source: §4.1 · p. 5
4.2 Spectral Rank Utilization
- Normalized hard-rank utilization declines with width while soft-rank utilization stays nearly scale-invariant, so the core occupies a shrinking share as the tail keeps pace.The reported hard-utilization slopes are around −0.5 across the three scales, while soft-utilization slopes are approximately zero.Source: §4.2 · p. 5
- Utilization distinguishes dilution, where soft utilization stays flat or rises as hard utilization falls, from collapse, where both decline, especially at large D for 250M.The reported patterns are consistent across backbones and independent of absolute width.Source: §4.2 · p. 5
- Effective dimension grows sublinearly and remains a small fraction of width; its marginal gain per unit width drops beyond approximately 2.67×–4×.The authors suggest LLaMA can stop nearer 2.67× for parameter efficiency, while families targeting stronger tail expressivity may push to 4×.Source: §4.2 · p. 6
- Layerwise eDim trends motivate pruning or narrowing persistently low-eDim layers while widening layers whose eDim continues to rise, rather than scaling uniformly.This is the authors’ proposed non-uniform width-allocation strategy.Source: §4.2 · p. 6
4.3 Scaling Laws for Spectral Concentration
- Concentration rises from around 0.57 to nearly 0.99 as α increases from 0.8 to 2.0; around α≈1.2, spectra shift from dilution to collapse.Below that transition, at least half the spectrum is needed to explain 80% of variance; above it, fewer than 10% of directions suffice.Source: Table 3 · p. 7, §4.3 · p. 7
4.4 Spectral Scaling Dynamics
- By approximately 5K training steps, width-scaling curves stabilize with R2≥0.6; normalized hard-rank exponents are negative while soft-rank exponents are positive.Transient width crossovers can occur, but the reported summary says they do not alter the converged exponents or their trade-off.Source: §4.4 · pp. 7–8
5 Case Study for Spectral Rank
Across the case studies, FFN width does not translate uniformly into spectral capacity: normalization changes how tail and dominant modes grow, and can prevent collapse at wider widths. Weight normalization and hyperspherical normalization stabilize Post-LN training, but their spectral utilization and perplexity outcomes differ.
5.1 LayerNorm and Spectral Rank
- Pre-LN soft-rank slopes are β ≈0.88 at 70M and β ≈1.07 at 130M, while hard-rank slopes are β ≈0.45/0.60; Post-LN soft-rank slopes fall to ∼0.71 −0.82.Post-LN hard-rank slopes rise modestly to ∼0.52 −0.56, remaining sublinear; the 250M Post-LN results are unavailable in Table 4 because of instability at higher widths.Source: §5.1 · p. 8, Table 4 · p. 9
- Mix-LN combines near-linear soft-rank scaling (β ≈0.97 − 1.10) with hard-rank growth of β ≈0.59 −0.63, above Pre-LN and Post-LN levels.The passage characterizes this as preserving tail coverage while improving dominant-mode scaling.Source: §5.1 · p. 8
5.2 LLaMA-250M PostLN
- In LLaMA-250M PostLN, widening vanilla FFNs to 2.67d or 4d drives hard rank to ≲10^-3 and concentration to ≈1.0 within the first few thousand steps.The passage reports that most variance funnels into one or two directions, leaving most of approximately 3000 latent dimensions inactive, alongside sharply deteriorating performance.Source: §5.2 · pp. 8–9
- With Weight Norm., LLaMA-250M PostLN reaches perplexities of 25.1 at 2.67d and 24.3 at 4d, versus 27.1 for the vanilla 1d baseline.Its hard rank stabilizes at 10^-2–10^-1 and concentration at 0.25–0.3; Figure 6 shows smoother spectral dynamics with normalization.Source: Table 5 · p. 9, §5.2 · p. 9, Figure 6 · p. 9
5.3 Hyperspherical Normalization
- Hyperspherical normalization prevents collapse in LLaMA-250M PostLN, but its hard rank is about 30% below Weight Norm. and its perplexities are 27.9 at 2.67d and 26.5 at 4d.Its concentration is marginally higher, indicating a somewhat narrower effective basis than Weight Norm.Source: §5.3 · pp. 9–10
- At 2.67d, nGPT reaches perplexity 13.60 versus 14.07 for GPT-2 with GeLU, while its 1d perplexity is 15.01 versus the GPT-2 baseline of 15.63.The nGPT configuration combines SwiGLU with hyperspherical weight and activation normalization and a learnable residual eigen-learning rate.Source: §5.3 · p. 10
- Across the GPT-2 width sweep, nGPT narrows the soft-hard rank gap and sustains dominant-mode growth while soft-rank tails expand without overwhelming the spectrum.Normalized utilization remains uneven in GPT-2 but flattens into near-straight lines in nGPT.Source: §5.3 · pp. 10–11, Figure 7 · p. 10
6 Conclusion
The conclusion frames FFN width selection as a spectral-utilization problem: widening tends to increase soft-rank utilization while hard-rank utilization grows sublinearly. It identifies normalization placement as a moderator of these dynamics and points to layer-wise scheduling and pruning as width-efficient design directions.
6 Conclusion
- Widening follows a tail-first pattern: soft-rank utilization remains near-linear while hard-rank utilization declines, producing spectral dilution and spectral collapse that limit naive width growth.Pre-LN amplifies tails, Post-LN suppresses them, and Mix-LN balances both.Source: §6 · p. 11
- The authors identify layer-wise scheduling and pruning as directions for width-efficient designs.Source: §6 · p. 11
Limitations
The study’s scope is limited to English decoder-only models up to 250M parameters.
Limitations
- The study covers English decoder-only models up to 250M parameters and does not validate spectral behavior in multilingual or encoder-decoder settings.The authors also note that spectral metrics correlate with perplexity without establishing causality, scalar metrics such as SUI may need finer-grained subspace analysis, and eigencomputations could be challenging at extreme scales.Source: Limitations · p. 11