PaperlayerView exact revision on arXiv

Source-linked AI summary

NerVE: Nonlinear Eigenspectrum Dynamics in LLM Feed-Forward Networks

Nandan Kumar Jha, Brandon Reagen

arXiv:2603.06922v2cs.LG

TL;DR

FFN nonlinear dynamics in LLMs remain difficult to characterize despite their central computational role. NerVE tracks eigenspectrum changes with four metrics and finds that nonlinearities reinject variance across latent directions, with optimizer geometry modulating this effect.

  • Problem

    FFN nonlinearities remain underexplored, and existing tools lack systematic, efficient characterization of high-dimensional latent structure and variance redistribution.

  • Method

    NerVE is a unified, memory-efficient framework that tracks pre- and post-activation FFN eigenspectra using four scale-invariant spectral metrics.

  • Results

    Across settings, FFN nonlinearities reinject variance across eigenmodes, increasing spectral entropy and participation ratio while reducing top-heaviness.

  • Takeaways & Limitations

    NerVE indicates that nonlinearities regulate latent geometry, while optimizer geometry modulates whether they repair spectral collapse or refine well-conditioned spectra.

  • Takeaways & Limitations

    NerVE metrics do not directly predict downstream task quality, and full eigendecompositions can be costly in large dimensions.

Abstract

from arXiv · show

We introduce NerVE, a unified eigenspectral framework for understanding how feed-forward networks (FFNs) in large language models (LLMs) organize and regulate information flow in high-dimensional latent space. Despite FFNs dominating the parameter budget, their high-dimensional dynamics remain poorly understood. NerVE addresses this gap through lightweight, memory-efficient tracking of eigenspectrum dynamics via four complementary metrics: Spectral Entropy (dispersion), Participation Ratio (effective dimensionality), Eigenvalue Early Enrichment (top-heaviness), and Jensen-Shannon divergence (distributional shifts). Our key insight is that FFN nonlinearities reinject variance across eigenmodes, fundamentally governing latent dimension utilization, and that optimizer geometry strongly modulates the extent of this variance reinjection. We validate NerVE across model scales, and diverse architectural and optimizer configurations, each uniquely shaping FFN dynamics: normalization schemes controlling variance flow; FFN weight geometries constraining latent space; positional encoding and activation functions regulating information flow; and optimizer choices redistributing effective capacity across depth. Across these settings, NerVE consistently recovers stable spectral signatures that correlate with model's generalization ability and respond predictably to design choices, generalizing beyond transformer to MLP-Mixer architectures, providing actionable insights for architectural and optimizer choices beyond trial-and-error.

1 INTRODUCTION

NerVE addresses the limited understanding of FFN nonlinear dynamics by providing an online, memory-efficient eigenspectral framework for tracking latent geometry. Across architectural and optimizer settings, it shows that nonlinearities reinject variance across eigenmodes and reorganize information flow.

  • Motivation: FFNs dominate transformer parameter and computational budgets, yet their high-dimensional nonlinear transformations remain underexplored and difficult to interpret.These transformations reorganize, compress, and propagate information extracted by attention across layers.
  • Framework: NerVE provides unified, online, memory-efficient eigenspectrum analysis using spectral entropy, participation ratio, eigenvalue early enrichment, and Jensen-Shannon divergence.The metrics summarize pre- and post-activation spectra through dispersion, effective dimensionality, top-heaviness, and distributional shifts.
  • Evaluation scope: NerVE is applied across normalization schemes, FFN weight geometries, positional encodings, optimizers, model scales, and MLP-Mixer architectures.The framework is presented as a way to compare how these design choices shape FFN dynamics.
  • Core finding: FFN nonlinearities actively reinject variance across eigenmodes and reactivate under-utilized directions rather than merely rescaling activations.In GPT-2, post-activation spectra show increased SE and PR, decreased EEE, and depth-localized JS transition bands.
  • Core finding: Optimizer geometry modulates variance reinjection, shifting FFN nonlinearity between repairing spectral collapse and refining a well-conditioned spectrum.This distinction is stated as a central conceptual contribution of NerVE.

2 NERVE: A PRINCIPLED FRAMEWORK FOR EIGENSPECTRUM ANALYSIS

NerVE analyzes FFN information structure and propagation by tracking covariance eigenspectra through activation collection, covariance computation, eigendecomposition, and four spectral metrics. The framework characterizes variance dispersion, effective dimensionality, top-heaviness, and pre-to-post nonlinear distributional shifts.

  • Framework pipeline: NerVE comprises activation collection, covariance computation, eigendecomposition, and spectral-metric calculation to analyze FFN latent-space information structure and propagation.The framework examines variance distribution, nonlinear reshaping within layers, and evolution across layers and training.
  • Activation collection: PreAct and PostAct activations are collected from non-gated or gated FFNs, including SwiGLU-style architectures.For non-gated FFNs, PreAct is the up-projection output and PostAct is the activated input to the down projection; gated FFNs use element-wise multiplication of gated and up-projected paths.
  • Covariance estimation: Activations are flattened from [B, S, D] to [B × S, D], treating all N = B × S tokens as independent samples while discarding sequence order.This produces an unbiased covariance matrix across batch tokens in FFN latent space; using all tokens avoids subsampling approximations.
  • Spectral metrics: NerVE computes Spectral Entropy, Participation Ratio, Eigenvalue Early Enrichment, and Jensen-Shannon divergence from each covariance eigenspectrum.Spectral Entropy measures eigenvalue-distribution uniformity, Participation Ratio measures effective dimensionality, and Eigenvalue Early Enrichment measures top-heaviness.
  • Eigenvalue Early Enrichment: In a 768-dimensional latent space, the One dimension spectrum has an EEE of 1.00, indicating nearly all variance is concentrated in one dominant principal component.EEE values near 1 indicate top-heavy concentration, whereas values near 0 correspond to nearly uniform spectra.
  • Jensen-Shannon divergence: Jensen-Shannon divergence measures distributional shifts between pre- and post-activation eigenspectra caused by FFN nonlinearity.Unlike the other metrics, JS compares two eigenspectra within a layer rather than describing one eigenspectrum in isolation.

3 EXPERIMENTAL RESULTS

Across architectures, normalizations, widths, positional encodings, and optimizers, NerVE metrics reveal how FFN nonlinearities redistribute variance, reshape eigenspectra, and govern latent-space utilization. These spectral signatures also track generalization and support online diagnosis and architectural ranking.

  • Nonlinearity and eigenspectrum dynamics: FFN nonlinearities increase Spectral Entropy and Participation Ratio while reducing Eigenvalue Early Enrichment, redistributing variance across more latent directions.Pre-activation spectra are top-heavy with lower SE and PR; post-activation spectra show higher SE and PR and lower EEE.
  • Activation functions: GELU and ReLU share variance reinjection, spectral flattening, and distributional reordering, but ReLU stabilizes SE and PR earlier while GELU ultimately reaches higher PRpost.The passage characterizes these differences as distinct pace and extent of the same qualitative trajectory.
  • Normalization schemes: Without LayerNorm, early GELU FFNs exhibit spectral inertia with EEEpost ≈1 and JS ≈0, whereas ReLU variants produce ≈20× to 300× PR gains in the first two layers.Both ReLU variants reduce the perplexity gap to the LayerNorm baseline by ≈50%.
  • Parametric normalization: Spectral normalization yields smooth flattening with lowest EEE post ≈-0.45 and PR post ≈200, while hyperspherical normalization overshoots PR post beyond 600 and retains high EEE post.SNorm’s flattening is reflected in uniformly negative ∆EEE values, especially in deeper layers.
  • Diagnostics and optimizer geometry: NerVE metrics correlate with validation loss at |r| ≥0.97 across widths, while optimizer geometry differentiates spectral burden: Muon preserves near-isotropic spectra and AdamW induces early-layer collapse.RoPE achieves evaluation perplexity 15.20 vs 16.78 for NoPE, and short NerVE runs can rank configurations before convergence.

4 RELATED WORK

Prior work uses spectral signals from weights or representations to analyze model internals, predict accuracy, quantify compression, and study attention localization. NerVE instead tracks FFN eigenspectrum dynamics to diagnose how nonlinearities redistribute latent-space variance and inform architectural and optimizer choices.

  • 4 RELATED WORK: RankMe and Diff-eRank use spectral-entropy Rank measures to predict downstream accuracy and quantify compression.
  • 4 RELATED WORK: Spectral concentration in QK weight matrices has been linked to attention localization, with one-step belief-propagation refinement proposed to address it.
  • 4 RELATED WORK: NerVE directly tracks FFN eigenspectrum dynamics, revealing nonlinear variance redistribution in latent space and diagnosing architectural and optimizer choices.

5 LIMITATIONS AND CONCLUSION · N Discussion: Why Top-heaviness Over Tail-heaviness

NerVE’s eigenspectral metrics provide complementary views of high-dimensional FFN variance usage but do not directly predict downstream task quality. Applying them at large dimensions can also be computationally costly, requiring sampling or approximation.

  • 5 LIMITATIONS AND CONCLUSION: NerVE eigen-metrics analyze how FFNs organize variance in LLMs but do not directly predict downstream task quality.This limitation is discussed alongside further framework limitations in Appendix M.
  • 5 LIMITATIONS AND CONCLUSION: Full eigendecompositions become costly in large dimensions.The computational burden often necessitates sampling or approximation, as detailed in Appendix G.
  • 5 LIMITATIONS AND CONCLUSION: Sampling or approximation can be necessary when computing eigendecompositions at large scale.The passage identifies this as a practical response to the cost of full eigendecompositions.
  • 5 LIMITATIONS AND CONCLUSION: Despite these constraints, each NerVE metric contributes a distinct view of high-dimensional usage.The metrics are presented as complementary rather than redundant.
  • 5 LIMITATIONS AND CONCLUSION: The metrics together provide complementary perspectives on how high-dimensional FFN variance is used.Their complementarity is the main positive conclusion accompanying the stated limitations.
  • 5 LIMITATIONS AND CONCLUSION: The framework’s limitations warrant consulting Appendix M for further details.Appendix G is specifically referenced for eigendecomposition sampling or approximation.

A IMPLEMENTATION DETAILS AND METHODOLOGICAL CONSIDERATIONS … B.1 METRICS DESIGN AND JUSTIFICATIONS

NerVE computes paired pre- and post-activation eigenspectra from token-level FFN activations and uses four complementary, scale-invariant metrics to diagnose dispersion, dimensionality, top-heaviness, and nonlinear distributional restructuring. Its methodological design emphasizes reproducible covariance estimation, numerical efficiency, bounded comparisons, and complementary sensitivity across the eigenspectrum.

  • A.1 FRAMEWORK OVERVIEW: NerVE collects pre- and post-activation FFN representations, flattens tokens into X ∈ R^(B×S)×D, and computes covariance eigenvalues for each layer.Pre-activations are captured after Wup and post-activations before Wdown; eigenvalues are computed from mean-centered activations.
  • A.1 FRAMEWORK OVERVIEW: The framework computes SE for dispersion, PR for effective dimensionality, EEE for top-heaviness, and JS for pre/post spectral distributional shifts.JS captures geometric restructuring performed by the nonlinearity, while SE, PR, and EEE characterize individual spectra.
  • A.2 IMPLEMENTATION DETAILS FOR COMPUTING COVARIANCE MATRICES: Eigenvalues are sorted in descending order before metric computation, which is required for EEE’s cumulative variance calculation.Tokens are treated as independent samples after flattening batch and sequence dimensions, with covariance Σ = 1/(N−1) X̂^⊤X̂.
  • A.2 IMPLEMENTATION DETAILS FOR COMPUTING COVARIANCE MATRICES: Pre- and post-activation covariances use the identical N-token population, ensuring JS compares nonlinear transformations on matched samples.The same token subset must be retained when sub-sampling is used.
  • A.2 IMPLEMENTATION DETAILS FOR COMPUTING COVARIANCE MATRICES: The implementation uses float32 tensors, ε = 1e−12 stabilization, sequential layer processing, and torch.linalg.eigvalsh for efficient, stable eigendecomposition.Sequential processing bounds peak GPU memory by one layer’s covariance pair rather than all L layers.
  • B EIGENSPECTRUM METRICS: DESIGN PRINCIPLES AND DIAGNOSTIC GUIDE: Table 4 summarizes each metric’s input, bounds, and captured property, while SE, PR, and EEE describe single spectra and JS compares pre- and post-activation shapes.All four metrics are invariant to uniform eigenvalue scaling.
  • B.1 METRICS DESIGN AND JUSTIFICATIONS: The metric set is complementary because SE and PR can coincide for spectra with different variance allocations, whereas EEE resolves this ambiguity through cumulative variance.SE is more responsive to mid-to-tail modes, PR is driven primarily by dominant modes, and EEE approaches 1 when a few large eigenvalues dominate.
  • B.1 METRICS DESIGN AND JUSTIFICATIONS: JS is required to measure nonlinear geometric restructuring, with large JS indicating variance redistribution and small JS indicating rescaling of existing directions.JS is symmetric and bounded by 0 ≤ JS(Ppre ∥ Ppost) ≤ ln(2), supporting unbiased, standardized comparisons.

B.2 METRIC RELATIONSHIPS AND FAILURE MODES · B.3 DECOMPOSING NONLINEARITY-INDUCED EIGENSPECTRUM RESTRUCTURING · C EIGENSPECTRAL SIGNATURE

NerVE requires joint interpretation of its eigenspectral metrics because they can agree, diverge, or mislead when used individually. Cross-activation measures further distinguish dimensional expansion, top-heaviness reduction, and spectral change, showing that nonlinear effort and sustained flattening—not isolated metric values—matter for outcomes.

  • B.2 METRIC RELATIONSHIPS AND FAILURE MODES: SE and PR often co-increase while EEE decreases and JS falls across depth, indicating variance reinjection, latent-space utilization, and less geometric restructuring in deeper FFNs.This aligned pattern is observed during standard training of well-configured models such as GPT-2 PreLN with GELU.
  • B.2 METRIC RELATIONSHIPS AND FAILURE MODES: Metric divergence localizes spectral change: SE can rise with stable PR, PR can rise alongside EEE, or SE and PR can rise while JS remains near zero.These patterns respectively indicate tail reshaping, unequal activation of new directions, or flattening without meaningful nonlinear restructuring.
  • B.2 METRIC RELATIONSHIPS AND FAILURE MODES: PR alone can mislead: HNorm exceeds 600 post-activation yet has worse perplexity than Spectral normalization because its EEE post remains high across depth.The passage attributes this to ∆EEE flattening that does not persist.
  • B.2 METRIC RELATIONSHIPS AND FAILURE MODES: PR at FFN width D = 1d is constrained to a narrow range, so values in bottleneck regimes should be normalized by D before comparison with wider configurations.This limitation concerns direct cross-configuration comparison rather than spectral quality itself.
  • B.2 METRIC RELATIONSHIPS AND FAILURE MODES: EEE saturates near its upper bound in highly anisotropic spectra, whereas PR remains informative by continuing toward 1 as concentration increases.SE is also insensitive to absolute scale because it uses normalized eigenvalues, while PR is preferred when total variance matters.
  • B.3 DECOMPOSING NONLINEARITY-INDUCED EIGENSPECTRUM RESTRUCTURING: JS measures total spectral change, PR(Post/Pre) measures dimensional expansion, and ∆EEE = EEE post −EEE pre measures top-heaviness reduction.More negative ∆EEE indicates stronger flattening, and the three quantities are complementary rather than redundant.
  • B.3 DECOMPOSING NONLINEARITY-INDUCED EIGENSPECTRUM RESTRUCTURING: AdamW produces the largest PR gains and highest JS but the lowest PRpost, whereas Muon achieves the highest PRpost with the smallest PR gains and lowest JS.AdamW’s collapsed pre-activation spectrum makes nonlinearities spend capacity on corrective repair, while Muon preserves a well-conditioned spectrum.
  • B.3 DECOMPOSING NONLINEARITY-INDUCED EIGENSPECTRUM RESTRUCTURING: High PRpost alone is insufficient: HNorm attains larger PRpost yet underperforms, while SNorm sustains lower EEEpost and achieves the best perplexity among weight-normalization methods.The distinguishing factor is sustained flattening throughout training.

C.1 SPECTRAL SIGNATURE OF LAYERNORM POSITIONING: PRELN, MIXLN, AND POSTLN … G TOKEN SUB-SAMPLING AND LOW-RANK APPROXIMATION

Across LayerNorm placements and model scales, eigenspectral signatures track FFN latent-space utilization and align with perplexity, while larger-scale behavior can include instability. The merged sections also examine positional encoding, FFN width, normalization-free variants, and efficient eigenspectrum approximations.

  • C.1 SPECTRAL SIGNATURE OF LAYERNORM POSITIONING: PRELN, MIXLN, AND POSTLN: Lower-perplexity LayerNorm configurations generally exhibit higher post-activation spectral entropy and participation ratio, indicating greater FFN latent-space utilization.Across GPT2-125M, LLaMA-70M, and LLaMA-130M, spectral-signature rankings corroborate evaluation-perplexity rankings.
  • C.2 SPECTRAL SIGNATURE IN LARGER LLAMA MODELS: At larger scales, MixLN improves LLaMA-250M perplexity to 24.2 versus 24.5 for PreLN, but destabilizes LLaMA-1.3B after 7K steps, where perplexity reaches 1457.1 versus 21.2 for PreLN.In LLaMA-250M, MixLN avoids PreLN’s mid-network spectral collapse; in LLaMA-1.3B, PreLN maintains stable spectral entropy and higher deep-layer participation ratio.
  • C.2 SPECTRAL SIGNATURE IN LARGER LLAMA MODELS: LayerNorm placement significantly influences FFN latent-space utilization, with Pre-LN configurations preserving and amplifying feature diversity especially in deeper layers.The reported MixLN gains suggest partial PreLN use can balance stability and expressivity.
  • C.3 SPECTRAL SIGNATURE OF POSITIONAL ENCODING: NOPE VS ROPE: Figure 13 compares RoPE with NoPE using layerwise post-activation spectral entropy and Eigenvalue Early Enrichment values.The comparison uses GPT-2 models trained with 512-token context on 26B tokens from OpenWebText.
  • NoPE (SE) and RoPE (SE): The NoPE and RoPE analyses separately track spectral entropy over training through the corresponding SE plots.The supplied passages identify the NoPE (SE) and RoPE (SE) subsections but provide no numerical comparison.
  • D LAYERNORM POSITIONING AND FFN WIDTH SWEEP: At FFN width D = 6144, PreLN sustains PRpost ≈1822 effective dimensions versus 71 for PostLN, a 25× difference in absolute latent capacity.MixLN remains closer to PostLN across the width sweep, while normalized PRpost can understate the absolute gap.
  • E SPECTRAL SIGNATURE ACROSS FFN WIDTH SWEEPS BASELINE MODELS: Width-sweep experiments evaluate eigen metrics and their Pearson correlations with evaluation loss in baseline GPT-2 models using GELU and ReLU activations.The corresponding figures cover widths D = 1d to 8d.
  • F SPECTRAL SIGNATURE ACROSS FFN WIDTH SWEEPS IN NORM-FREE: Normalization-free GPT-2 width sweeps similarly measure spectral entropy, participation ratio, and Pearson correlations with evaluation loss for GELU and ReLU.The figures span widths D = 1d to 8d.

G.1 EIGEN-METRIC FIDELITY AND DIAGNOSTIC VALIDITY UNDER APPROXIMATIONS … H.2 MEMORY-EFFICIENT EIGENSPECTRUM ANALYSIS

NerVE’s approximations preserve eigenspectrum metrics and pre-activation diagnostics best under token sub-sampling, while low-rank truncation causes stronger correlation distortions. Its efficient implementation keeps computational overhead modest and GPU memory bounded through sampling, eigenvalue-only computation, sequential processing, and hybrid storage.

  • G.1 EIGEN-METRIC FIDELITY AND DIAGNOSTIC VALIDITY UNDER APPROXIMATIONS: At 10% token sampling, the worst-case metric error is 10.24% for PR post, while SE and EEE errors remain below 2%.Low-rank truncation introduces substantial bias because the metrics aggregate information across the full spectrum.
  • G.1 EIGEN-METRIC FIDELITY AND DIAGNOSTIC VALIDITY UNDER APPROXIMATIONS: Across 5%-50% sampling, SEpre remains |r| > 0.97 and PRpre remains |r| > 0.91 with validation loss.Post-activation correlations are less stable: SEpost reaches |r| ≈0.34–0.38, PRpost falls below |r| < 0.24, and PRpost flips sign at 25%.
  • G.1 EIGEN-METRIC FIDELITY AND DIAGNOSTIC VALIDITY UNDER APPROXIMATIONS: Low-rank methods weaken diagnostic correlations more severely than token sub-sampling, especially for post-activation metrics.Lanczos-512 reaches SEpre |r| = 0.96 and PRpre |r| = 0.90, whereas RandSVD yields SEpre |r| < 0.18 and PRpre |r| < 0.05.
  • H.1 COMPUTATIONAL COMPLEXITY AND MEMORY OVERHEADS OF NERVE: At every 1000 steps, full-batch eigendecomposition takes 14.41s per logging step and adds 1.3% training overhead.At 5% token sampling, wall-clock time falls to 9.89s; at every 200 steps, full-batch overhead is 6.4%.
  • H.2 MEMORY-EFFICIENT EIGENSPECTRUM ANALYSIS: Memory optimization restricts GPT-2 peak GPU usage to approximately 2 × 36MB per layer instead of accumulating 2 × 12 × 36MB = 864MB across all FFNs.The approach processes layers sequentially with cleanup between computations.
  • H COMPUTATIONAL AND MEMORY EFFICIENCY OF NERVE: NerVE computes eigenvalues without eigenvectors, processes layers sequentially, and stores activations on CPU while computing eigenvalues on GPU.Metrics SE, PR, EEE, and JS are computed from the eigenvalues before immediate activation cleanup.

I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE · J TOKEN-POSITION EFFECTS ON FFN EIGENSPECTRUM · K FFN EIGENSPECTRUM DYNAMICS: LAYERNORM VS RMSNORM

Across MLP-Mixer and GPT-2, NerVE shows that FFN nonlinearities reshape eigenspectrum geometry and that this reshaping depends on activation placement and token position. MLP-Mixer exhibits weaker spatial-position effects than GPT-2, while normalization-free GPT-2 largely suppresses position dependence.

  • I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE: MLP-Mixer extends eigenspectrum analysis beyond transformers by replacing self-attention with functionally analogous channel-mixing FFNs and enabling evaluation on vision tasks.The channel-mixing MLPs expand and squeeze latent representations through nonlinearities, supporting cross-modal analysis of FFN dynamics.
  • I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE: Across all activation settings, post-activation spectra have higher spectral entropy and participation ratio than pre-activation spectra, with gaps widening during the first 20–40 epochs.This indicates that Mixer nonlinearities expand effective FFN dimensionality rather than merely reshuffling eigenmodes.
  • I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE: ReLU in channel-mixing FFN2 produces larger post-activation SE/PR and lower EEE than GELU, yielding flatter spectra through variance redistribution away from leading eigenmodes.Changing FFN2 has a stronger effect than changing the token-mixing FFN1.
  • I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE: The GELU-FFN1/ReLU-FFN2 configuration has the highest post-activation PR and lowest post-activation EEE, while maintaining small JS divergence across most layers.This configuration most aggressively flattens spectra and expands effective latent dimensionality.
  • I EIGENSPECTRUM DYNAMICS IN A NON-TRANSFORMER ARCHITECTURE: Activation choice shifts JS divergence toward early layers with GELU in FFN2 and toward deeper layers with ReLU in FFN2, while interior layers remain relatively stationary.The result suggests that activation mainly changes how boundary layers reshape the eigenspectrum.
  • J TOKEN-POSITION EFFECTS ON FFN EIGENSPECTRUM: In GPT-2, token positions have comparable pre-activation PR but diverge after nonlinearity, with late-token post-activation PR roughly twice early-token PR under GELU.The separation emerges mainly in deeper FFN layers and remains visible under ReLU.
  • J TOKEN-POSITION EFFECTS ON FFN EIGENSPECTRUM: Removing LayerNorm nearly eliminates GPT-2 position-dependent FFN geometry, whereas MLP-Mixer shows only small and nearly coincident top-versus-bottom patch PR differences.Normalization-free GPT-2 position groups nearly collapse at both pre- and post-activation stages across depths.

K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2 … L.1 ADAMW VS MUON VS DION

The ablations show that FFN eigenspectrum dynamics are driven more by activation, normalization placement, and optimizer geometry than by LayerNorm versus RMSNorm. Across GPT-2 and MLP-Mixer, Muon preserves higher-dimensional pre-activation spectra, while LayerNorm and RMSNorm remain qualitatively similar.

  • K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2: In GPT-2, activation type and normalization placement produce larger post-activation PR shifts than swapping LayerNorm for RMSNorm.LayerNorm and RMSNorm distributions substantially overlap within each activation-placement pair.
  • K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2: GELU gives the largest PR values, ReLU compresses the spectrum, and learnable leaky ReLU lies between them with a narrower spread.Changing PreLN to PostLN or MixLN also clearly affects PR, whereas LayerNorm-to-RMSNorm changes are second-order.
  • K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2: Layerwise FFN eigenspectral structure is quantitatively robust across LayerNorm and RMSNorm, with GELU and ReLU correlations of R2 ≈0.14 and R2 ≈0.17.High-PR and low-PR layers tend to retain their ordering across normalization schemes.
  • K.1 LAYERNORM VS RMSNORM ABLATION ON GPT-2: NerVE’s centered FFN activations remove the DC component for both normalization schemes, supporting normalization-independent eigenspectrum conclusions.This centering occurs during covariance construction, regardless of whether the model itself uses LayerNorm or RMSNorm.
  • K.2 LAYERNORM VS RMSNORM ABLATION ON MLP-MIXER: In MLP-Mixer trained on CIFAR-100, LayerNorm and RMSNorm produce similar SE, EEE, and converged layerwise JS dynamics.The comparison tracks post-activation SE, PR, EEE, and final-epoch JS across training.
  • K.2 LAYERNORM VS RMSNORM ABLATION ON MLP-MIXER: LayerNorm’s final-phase PR post advantage over RMSNorm coincides with lower EEE post and accuracy of 66.96% vs 66.38%.The main spectral differences emerge late in training, while the overall dynamics remain qualitatively stable.
  • L.1 ADAMW VS MUON VS DION: Across context lengths 512 and 1024, AdamW shows large early PR gains, high JS, and relatively high EEE post, whereas Muon shows the smallest PR gains, lowest JS, and lowest EEE post.These patterns indicate AdamW-induced pre-activation collapse followed by aggressive but incomplete nonlinear repair, while Muon maintains flatter post-activation spectra.
  • L.1 ADAMW VS MUON VS DION: Muon consistently has the highest PR pre across layers, and increasing context length preserves this ordering while intensifying early-layer collapse under AdamW and Dion.Muon therefore produces well-conditioned, activation-compatible pre-activation representations across sequence lengths.

L.2 ADAMW VS ADAFACTOR · L.3 ADAMW VS SGD

Across GPT-2, Adafactor preserves the qualitative FFN spectral effects seen with AdamW while producing stronger spectral expansion, higher latent-space utilization, and deeper-layer flattening, especially in normalization-free ReLU models. In MLP-Mixer, SGD achieves higher effective dimensionality and better FFN capacity utilization than Adam, alongside superior accuracy.

  • L.2 ADAMW VS ADAFACTOR: Under Adafactor, FFN nonlinearities increase SE and PR while lowering EEE, transforming top-heavy, anisotropic pre-activation spectra into flatter post-activation spectra.This reproduces the qualitative role of nonlinearities previously observed under AdamW.
  • L.2 ADAMW VS ADAFACTOR: Adafactor consistently produces higher post-activation PR than AdamW across four GPT-2 configurations, with the largest gap in normalization-free ReLU.The comparison covers PreLN and normalization-free models with GELU and ReLU activations.
  • L.2 ADAMW VS ADAFACTOR: Adafactor drives stronger spectral expansion than AdamW, activating more FFN latent capacity, particularly in the normalization-free ReLU configuration.Thus, optimizer geometry modulates the magnitude of nonlinear variance reinjection without changing its qualitative direction.
  • L.2 ADAMW VS ADAFACTOR: Deeper GPT-2 layers rapidly reach and maintain higher PR post throughout training, with this effect consistently stronger under Adafactor than AdamW.The advantage is especially evident in deeper FFNs and normalization-free ReLU models.
  • L.2 ADAMW VS ADAFACTOR: Both AdamW and Adafactor reduce ∆EEE below zero, while Adafactor reaches more negative ∆EEE in deeper layers, indicating stronger suppression of top eigenvalues.More negative ∆EEE corresponds to a flatter spectrum and more aggressive spectral flattening.
  • L.3 ADAMW VS SGD: On MLP-Mixer, SGD outperforms Adam, achieving 68.07% versus 66.96% accuracy.The experiment uses the GELU MLP-Mixer baseline on CIFAR-100, with SGD configured at learning rate 5e-2, momentum 0.9, and weight decay 1e-4.
  • L.3 ADAMW VS SGD: SGD attains substantially higher post-activation SE and PR than Adam throughout MLP-Mixer training, indicating greater effective dimensionality and FFN capacity utilization.From epoch 10 onward, SGD shows superior SE post and PR post, particularly for the FFN2 latent space.
  • L.3 ADAMW VS SGD: Adam’s EEE remains near 1.0 throughout MLP-Mixer training, suggesting persistent concentration of variance in the top eigenvalues.This contrasts with SGD’s higher SE and PR trends, which indicate broader latent-space utilization.

M LIMITATIONS OF NERVE FRAMEWORK · N DISCUSSION: WHY TOP-HEAVINESS OVER TAIL-HEAVINESS

NerVE’s limitations concern missing cross-layer and position-specific spectral structure, computational costs, and approximation fidelity. It emphasizes top-heaviness over heavy-tail fitting because EEE directly captures dominant-subspace geometry while remaining stable and comparable across configurations.

  • M LIMITATIONS OF NERVE FRAMEWORK: NerVE computes spectra independently per layer and does not quantify cross-layer spectral coherence.Cross-layer measures such as overlap between consecutive layers’ top-k subspaces could test whether smooth spectral progression relates to healthier training dynamics.
  • M LIMITATIONS OF NERVE FRAMEWORK: Flattening [B, S, D] into [N, D] treats token positions as exchangeable to increase covariance sample size and target FFN-level geometry.This deliberate aggregation can mask substantial position-dependent structure revealed by position-stratified analysis.
  • M LIMITATIONS OF NERVE FRAMEWORK: Position-stratified analysis should complement aggregate metrics when position-specific FFN geometry is the target.
  • M LIMITATIONS OF NERVE FRAMEWORK: Full-batch covariance estimation and storing D×D matrices can be expensive at large scale.Token subsampling preserves much pre-activation diagnostic signal, whereas low-rank truncation can distort correlation-based analyses, especially for post-activation metrics; for D>10K, approximation fidelity should be validated.
  • N DISCUSSION: WHY TOP-HEAVINESS OVER TAIL-HEAVINESS: NerVE emphasizes top-heaviness because its goal is tracking how FFN nonlinearities reshape the dominant variance subspace from pre- to post-activation.Leading eigenmodes capture directions contributing most to second-moment energy and representation anisotropy, while EEE summarizes this dominant-subspace structure.
  • N DISCUSSION: WHY TOP-HEAVINESS OVER TAIL-HEAVINESS: Heavy-tail descriptors require sensitive tail-range selection and goodness-of-fit checks, complicating comparisons as FFN width D varies.Fixed k represents different spectral fractions, while fixed fractions represent different absolute spectral depths, requiring careful per-scale calibration.
  • N DISCUSSION: WHY TOP-HEAVINESS OVER TAIL-HEAVINESS: EEE is hyperparameter-free, numerically stable, and aligned with measuring dominant-subspace transformations and representational-capacity redistribution.The framework therefore considers tail-based measures but retains EEE as the primary metric for its central question.
Loading 2603.06922v2…