PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Heavy-Tailed Weight Spectra Predict Model Quality

Joseph Wilson, Chris van der Heide, Liam Hodgkinson, Zhichao Wang, Fred Roosta, Michael W. Mahoney

arXiv:2610.13123v1stat.MLcs.LG

TL;DR

Data-dependent DNN quality metrics can be costly or unavailable, while heavy-tailed weight spectra predict quality without a rigorous statistical account. This paper derives a final-layer-weight marginal-likelihood metric and an HTMP limit, finding predictive correlations across large CV models and LLMs and an optimal heaviness parameter.

  • Problem

    Data-dependent metrics may be expensive on massive datasets or unusable when foundation-model training data are undisclosed, while the statistical basis for heavy-tail weight-spectrum predictors remains unclear.

    Source: §1 · pp. 2–3

  • Method

    The authors derive a marginal likelihood for trained features that can be evaluated from final-layer weights, then analyze its deterministic limit under the HTMP spectral distribution.

    Source: §1 · p. 3

  • Results

    The final-layer Bayes free energy predicts quality across large CV models and LLMs, while the HTMP limit is convex in heaviness and has an optimal parameter empirically associated with over-fitting.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results provide a probabilistic connection between spectral properties and model quality, with the optimal HTMP heaviness empirically predicting over-fitting in large DNNs.

    Source: §1 · p. 3, §5 · p. 10

    The derived (BFE-W) is limited to final-layer weights and regression, and its dual form is not immediately comparable across models with differing activation dimension dL.

    Source: §5 · p. 10

Abstract

from arXiv · show

A solid understanding of the predictive success of deep neural networks (DNNs) remains elusive. For example, while many data-dependent metrics exist that seek to predict model quality of DNNs, these metrics may be expensive to compute for large datasets and/or may be impossible to compute when training datasets are not released. A practical and effective predictor of model quality involves analyzing DNN weight matrices, as prior work has established that heavy-tailed spectral distributions in trained weight matrices can be strongly predictive of model quality. However, such metrics lack a rigorous statistical justification. In this work, we aim to understand why these metrics perform well, from a statistical perspective. To do so, we employ a probabilistic framework to derive the marginal likelihood for a DNN, evaluated using only the final layer weights. Across large-scale convolutional DNNs and LLMs, we find that this marginal likelihood predicts model quality. To understand the role heavy-tailed spectra play in predictive performance, we derive the limiting log-marginal likelihood for spectra that follow the (heavy-tailed) High-Temperature Marchenko-Pastur (HTMP) distribution. We show that this limiting quantity is convex in a heaviness parameter, and we derive an optimal heaviness that empirically predicts over-fitting for large-scale DNNs.

1 Introduction

The paper addresses why heavy-tailed weight spectra predict DNN quality and develops a statistical account using a marginal likelihood evaluated from final-layer weights. It connects this metric and HTMP spectral parameters to model quality across large DNNs.

1 Introduction

  • Data-dependent quality metrics can be costly on massive datasets or unavailable for proprietary foundation-model training data, motivating weight-only predictors.Prior heavy-tail metrics predict quality, but their statistical justification is lacking.Source: §1 · pp. 2–3
  • The authors derive a Bayes free energy for trained features that can be evaluated using only final-layer weights and is strongly predictive of quality in large CV and language models.The relative metric uses the trace and log-determinant of WᵀW with regularization parameter λ.Source: §1 · p. 3
  • For fixed κ under HTMP, an optimal κ/β ratio exists, and the limiting free energy is strictly decreasing in κ.The paper identifies these spectral parameters as predictors of model quality.Source: §1 · p. 3
  • During training, free energy correlates with test error; for trained CV models κ/β relates to underfitting or overfitting, while LLMs attain the optimal ratio early.After that point, increasing κ during LLM training correlates with improved performance.Source: §1 · p. 3
  • The work derives a final-layer Bayes-free-energy formulation, evaluates its predictive correlation, and analyzes its HTMP limit and optimal parameters.It also empirically connects DNN training with optimization of HTMP parameters.Source: §1 · p. 3

2 Related work & background

The background situates the work among random-matrix and implicit-regularization approaches, then reviews heavy-tailed spectral metrics and the probabilistic training framework used in the paper.

RMT in deep learning

  • Random matrix theory studies limiting spectra and uses double-asymptotic limits, with applications to generalization and spectral laws in deep learning.The stated regime takes matrix dimensions m,n to infinity with m/n approaching a positive finite ratio.Source: §2 · p. 4

Implicit regularization

  • Implicit regularization can arise from architecture or optimization and may relate to generalization, but these metrics depend on training data.The passage notes both support and criticism for the proposed connection between small-batch SGD, flatter minima, and improved generalization.Source: §2 · p. 4

Heavy-tailed self-regularization (HT-SR)

  • HT-SR models trained-weight spectral tails with power-law or exponentially truncated power-law forms, and tail-heaviness metrics have predicted model quality.The cited CV finding is for within-group comparisons.Source: §2 · p. 4

Probabilistic training framework

  • The probabilistic training framework selects a parameter density minimizing expected loss plus a temperature-scaled divergence from the prior.The minimized objective is related to a PAC-Bayes bound when the loss is training loss, and to variational inference in SGD.Source: §2 · p. 4

3 Free energy for trained weights

The section derives a data-free Bayes free energy from trained weights and evaluates its relationship to model quality. The empirical results show correlations across CV and LLM models, with important scope limits for some architectures.

3 Free energy for trained weights

  • The section introduces the negative log-marginal for interpolating trained features as a quality measure evaluable from trained final-layer weights.It describes the quantity as correlating very well with quality for many large DNNs.Source: §3 · p. 5

3.1 Optimal features under interpolation for ridge regression

  • For interpolating ridge regression, optimal feature densities concentrate around regions of high marginal likelihood, which the authors compute for trained features.The marginal likelihood uses the stated prior on weights and the loss framework.Source: §3.1 · p. 5

3.2 Evaluation of free energy using final-layer weights

  • Under interpolation assumptions, the Bayes free energy for well-trained features can, as τ approaches zero, be evaluated from final-layer weights alone, up to terms canceling for same-dataset comparisons.The assumptions include n ≥ m, dL > m, rank conditions, and W = Φ†Y with ΦW = Y.Source: §3.2 · p. 5

3.3 Empirical evaluation

  • For ImageNet-1K torchvision models, (BFE-W) correlates with top-5 test accuracy at ρ = −0.71; LLM correlations are ρ = −0.66 on Pythia/LAMBADA and ρ = −0.77 on Qwen3.5/MMLU-Pro.The reported result is a strong correlation across groups for CV and strong correlations for both LLM suites.Source: §3.3 · p. 6
  • The empirical correlation does not hold for CV models using dropout before the final layer or for vision transformers.The authors suggest dropout prevents complex final-layer correlations and that the feature/weight split may be too crude for vision transformers.Source: §3.3 · pp. 6–7

4 Heavy-tails and free energy

The section connects trained weight spectra to model quality by deriving a limiting free energy under an HTMP spectral model and examining how its parameters track training dynamics. The analysis distinguishes the underfitting and overfitting patterns of computer-vision models from the training pattern observed in LLMs.

4.1 Trained weight spectra

  • After training, weight spectra are often modeled by the heavy-tailed HTMP distribution, unlike the Marchenko–Pastur spectra associated with initialization.The section formulates this as an asymptotic assumption for trained weight matrices as their dimensions grow.Source: §4.1 · p. 7

4.2 Limiting free energy

  • Under the HTMP limit, free energy is strictly convex and coercive in h, with unique h*=2γ/λ; at that optimum, increasing κ lowers it.Here h=κ/β; the decreasing-free-energy result is stated for τ→0 and β=β*(κ).Source: §4.2 · p. 8

4.3 Using HTMP to understand training dynamics

  • Empirical and asymptotic free energies very closely follow test error, with correlations ρ=0.927 and ρ=0.984; minimum test error occurs when h_t=h*.For CIFAR-10, the analysis uses a flattened third-layer convolutional weight because final-layer weights are small-dimensional.Source: §4.3 · p. 9
  • ρ=0.986 (Pythia-1B) and ρ=0.965 (Pythia-12B): free energies closely track test cross-entropy on LAMBADA.In these LLMs, β follows its optimal value while κ rises; the resulting free-energy decrease correlates with lower test error.Source: §4.3 · p. 9

5 Conclusion & discussion

The paper links heavy-tailed weight spectra to model quality through a data-free Bayes Free Energy metric and an HTMP limiting analysis. Its discussion identifies constraints on the derivation and on tuning the metric during computer-vision training.

5 Conclusion & discussion

  • The authors report that final-layer-weight Bayes Free Energy predicts model quality for computer-vision models and LLMs, while HTMP analysis yields an optimal heaviness that reflects training patterns.CV training correlates with optimizing h; LLMs reach h early, after which increasing κ correlates with improved performance.Source: §5 · p. 10
  • The Bayes Free Energy was derived only for final-layer weights and regression; extending it to models with pre-final-layer dropout or Vision Transformers remains future work.The authors note that those model types did not show correlation between the metric and model quality.Source: §5 · p. 10
  • For CV models during training, prior precision λ must be tuned post-hoc to optimize correlation.The authors identify deriving λ a priori as future work.Source: §5 · p. 10

A Derivation of feature marginal likelihood for ridge-regression

This appendix derives the marginal Gibbs likelihood for ridge regression with trained features, then defines Bayes Free Energy as its negative log. The derivation integrates over the weight matrix under a matrix-normal prior.

A Derivation of feature marginal likelihood for ridge-regression

A.1 Marginal likelihood derivation

  • The model writes Y=ΦW+ϵ, with matrix-normal weights and noise, and integrates out W using matrix-normal properties.The prior is centered at zero, and the noise covariance is given in the passage.Source: §A.1 · p. 17

A.2 Bayes free energy for regression

  • Bayes Free Energy is defined as the negative log marginal likelihood, with λm and τm set from the prior variance and noise parameter.The appendix says this yields the regression free-energy expression.Source: §A.2 · p. 17

B Derivation of BFE-W

The appendix derives primal and dual asymptotic expansions of the Bayes Free Energy for trained weights, then describes the HTMP spectral model used in the paper. The dual formulation has a comparison constraint across models with different activation dimensions.

B Derivation of BFE-W

  • The BFE-W derivation uses small-parameter asymptotic expansions of trace and log-determinant terms for singular matrices.The stated derivation relies on expansions for ν≪1.Source: B Derivation of BFE-W · p. 17

B.1 Primal form

  • Under rank and interpolation assumptions, the primal derivation expands trace and log-determinant terms as τ→0 using the trained weights.The proof relates W=Φ†Y to Φ=YW† and treats the two terms separately.Source: §B.1 · p. 18

B.2 Dual form

  • The dual expansion assumes dual interpolation; its free energy is not directly comparable across models with differing activation dimension dL.For models sharing a dataset, activation dimension, and full-rank features, the passage defines a relative dual free energy.Source: §B.2 · pp. 18–19
  • The dual free energy nevertheless correlates strongly with model quality across Pythia and Qwen3.5 models, even with differing activation dimensions.The passage leaves an asymptotic expansion enabling direct comparison across such dimensions for future work.Source: §B.2 · p. 19

C High-Temperature Marchenko-Pastur distribution

  • At finite κ in the high-temperature scaling, the beta-Laguerre ensemble’s limiting spectrum is HTMP; the cited scope excludes bulk-plus-spike spectra common in older DNNs.The limiting distribution is described for κ(N)→κ∈(0,∞) and N/D→γ∈(0,1).Source: C High-Temperature Marchenko-Pastur… · p. 20

C.1 Derivation of CDF for HTMP

The section derives the HTMP cumulative distribution function from its Stieltjes transform, expressing it through the continuous argument of a Tricomi function. It also gives a way to compute that argument using real and imaginary parts when a parameter is non-integer.

C.1 Derivation of CDF for HTMP

  • The HTMP CDF is obtained from the continuous argument of the Tricomi function U, with its argument unwrapped from zero.The derivation uses Stieltjes inversion and the fact that the HTMP density is supported on x ≥ 0.Source: §C.1 · p. 21
  • For non-integer 1 − κ/(2γ) + κ/2, the CDF computation can use real and imaginary parts of the boundary value of U instead of complex-valued functions.The resulting parts involve functions that can be computed numerically without complex arguments.Source: §C.1 · p. 21
  • The section gives expressions for those real and imaginary parts using Kummer’s function M and gamma functions.This result is restricted to the stated non-integer parameter case.Source: §C.1 · pp. 21–22

D Derivation of limiting free energy

Under spectral and integrability assumptions, the section derives the limiting free energy by evaluating the HTMP Shannon transform and first moment. It also establishes properties of the limiting free energy as the scale-based heaviness ratio varies.

D Derivation of limiting free energy

  • The limiting free energy is derived by combining the HTMP Shannon transform and first-moment limits under the stated spectral and integrability assumptions.The logarithmic-integrability assumption controls convergence of log det(WᵀW) as τ → 0.Source: D Derivation of limiting free energy · p. 22, §D.3 · p. 24

D.1 Shannon transform

  • The limiting log-determinant is given by the HTMP Shannon transform, which the section derives from the Stieltjes transform.The transform is defined as Vρ(x) = Eλ∼ρ log(1 + xλ), for x > 0.Source: §D.1 · pp. 22–23

D.2 First and second moments

  • For probability measures with finite first and second moments, the proof obtains the HTMPγ,κ,β first and second moments from the Stieltjes-transform expansion as z → −∞.Source: §D.2 · p. 23

D.3 Proof of Theorem 2.

  • The theorem’s limiting free-energy expression follows by combining the HTMP moment and Shannon-transform results.The derivation invokes Assumptions 2, 4, and 5.Source: §D.3 · p. 24

E Proofs for convexity and optimality of free energy

  • The next section studies the limiting free energy’s behavior with respect to κ and β, culminating in a proof of Theorem 3.Its stated results assume the preceding spectral and integrability conditions.Source: E Proofs for convexity and optimality… · p. 24

E.1 Optimality of free energy under ratio

The section characterizes an optimal scale-based heaviness ratio for the limiting free energy and relates the metric’s behavior to model quality across layers and model types. It also details precision-parameter choices and HTMP fitting procedures.

E.1 Optimality of free energy under ratio

  • For fixed κ, the limiting free energy is strictly convex and coercive in h = κ/β, yielding a unique optimal ratio.The Shannon transform and first moment increase with h; the optimal β is specified by the ratio h* = κ/β*.Source: §E.1 · pp. 24–25
  • The joint infimum occurs as κ and β grow without bound while κ/β approaches 2γ/λ; under this limit, HTMP approaches λ^-1 MPγ.The paper notes that MPγ generally does not interpolate complex training sets, making it mostly inaccessible for trained weights in sufficiently complex tasks.Source: §E.1 · p. 25

F.1 Connection of BFE-W to previous implicit regularization metrics

  • BFE-W’s trace term connects to HT-SR and Geometric Complexity, while its log-determinant measures functional complexity through the volume induced by last-layer weights.The log-determinant is also described as information gain for a Gaussian model with kernel WᵀW.Source: §F.1 · p. 27

F.2 Motivation for application of BFE-W for deeper layer weights

  • For deeper layers, BFE-W correlates well with test accuracy after training loss plateaus; ResNet50 correlations exceed 0.9, while Pythia QueryKey results are more mixed.Pythia Dense matrices correlate well across layers and model sizes.Source: §F.2 · p. 27

F.3 Choice of precision parameter λ

  • The constant λ choice works best for CV models, whereas both tested choices work for LLMs; dynamically setting λ assumes h is already optimal and removes CV’s under/overfitting behavior.For LLMs, h_t is approximately optimal, so the dynamic choice does not alter the correlation.Source: §F.3 · p. 27

F.4 Relation of optimal HTMP parameters to heavy-tailed results in literature

  • The paper’s normalized heaviness optimum parallels prior findings: λµ1 = 1, while an alternative normalization gives an optimum of 2.The authors relate these values to results on over-parameterized linear regression and DNNs, while noting that definitions and matrices differ.Source: §F.4 · p. 28

G.2 Fitting the HTMP

  • HTMP fitting fixes the aspect ratio and minimizes two-sided Kolmogorov–Smirnov distance, sometimes matching the empirical first moment directly.For some matrices, the top and bottom five eigenvalues are trimmed before fitting.Source: §G.2 · p. 28
  • When the specified HTMP parameter is an integer, computing the CDF requires complex numerical functions and is substantially slower.Otherwise, the procedure computes principal arguments and unwraps them across the eigenvalue grid.Source: §G.2 · pp. 28–29

H.1 Permuted label experiment

Across permuted-label, standard vision, and language-model experiments, free energy and HTMP parameters track model quality, but correlations across deeper weights vary by model. The experiments also show that optimal HTMP heaviness relates to underfitting and overfitting in different ways for vision and language models.

H.1 Permuted label experiment

  • ρ = 0.819: for permuted CIFAR-10, free energy closely predicts test error at interpolating epochs, although its absolute scale does not separate untrained from trained models.The authors loosely mark interpolation at 99% training accuracy; the analyzed matrix is a third-layer convolutional weight.Source: §H.1 · p. 29, Figure 6 · p. 30
  • After interpolation on permuted labels, the optimal model has an HTMP ratio h = κ/β closest to h*, even without feature learning.Source: §H.1 · p. 29

H.2 Extra ResNet and Pythia models

  • For ResNet18 and ResNet34, overfitting tracks h moving through h*, whereas Pythia models stay near h* and improve fit as κ increases.Source: §H.2 · p. 29

H.3 ESDs

  • The appendix supplies ESDs for ResNet50 on CIFAR-10 and Pythia-1B on Pile; Figure 10 shows the fitted HTMP distributions for those weights.Source: §H.3 · p. 29

H.4 Correlation of free energy with model quality across depth

  • ResNet50’s deeper weights on CIFAR-100 correlate very strongly with model quality, while Pythia results vary by matrix and model, including strong anti-correlation for Pythia-12B QueryKey weights.The authors caution that some care is needed when evaluating deeper weights.Source: §H.4 · p. 30
  • For Pythia, some models show excellent deeper-weight correlations across Dense and QueryKey layers, but Pythia-12B QueryKey weights show strong anti-correlation.The correlation analysis begins at epoch 43000 for Pythia to remove training-loss effects.Source: §H.4 · p. 30

I.1 Comparison to HT-SR metrics

Comparisons with HT-SR metrics show that predictive performance depends on model class and setting: BFE-W is consistent on updated non-dropout vision models but performs poorly for some dropout and vision-transformer models. Deeper weights can also be informative, though results vary across architectures and matrices.

I.1 Comparison to HT-SR metrics

  • Figure 7 tracks test cross-entropy, BFE-W, and Fλ∞ across epochs, while Figure 8 compares test error and free energy against fitted κ/β and κ.The caption reports strong training-time correlation for BFE-W; for CV, h* aligns with minimum error, while LLMs reach h* early and test error falls as κ increases.Source: Figure 7 · p. 31, Figure 8 · p. 31
  • For legacy ResNet, VGG, and DenseNet models, BFE-W and HT-SR metrics have similar, very strong correlations except for VGG, where dropout yields opposite across-group correlation; α and λ are poor predictors.Source: §I.1 · pp. 31–32
  • On updated torchvision models, BFE-W correlates very strongly with quality and is the most consistent predictor, while HT-SR metrics do not predict quality across classes or ResNet accuracy.The authors suggest, tentatively, that finely optimized weights may be over-regularized.Source: §I.1 · p. 32

I.2 When last-layer free energy does not predict model quality

  • Dropout before final-layer weights and vision transformers do not show good BFE-W correlation; the authors conjecture dropout may violate the features-and-weights decomposition or prevent complex final-layer correlations.Source: §I.2 · pp. 32–33
  • A significant proportion of ResNet50’s deeper weights show very strong correlation with model quality.Source: §I.2 · p. 33
  • For dropout models, HT-SR metrics are much stronger predictors than BFE-W despite only mild correlations (−0.42 ≥ρ ≥−0.66); HT-SR metrics are also poor predictors for vision transformers.The fitted α and λ do not show consistent correlation for these model classes.Source: §I.2 · p. 33
Loading 2610.13123v1…