PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix

Fangshuo Liao, Anastasios Kyrillidis

arXiv:2609.31983v1cs.LGmath.OC

TL;DR

The paper asks whether Hessian top-subspace stabilization is consistent across training setups and independent of parameter-update magnitude. It introduces an instability metric and identifies class-gradient structure linking Hessian and gradient-covariance subspaces. It reports stabilization in some settings and, under gradient flow, relates slow subspace evolution to separation between top and bulk Hessian eigenvalues.

  • Problem

    Prior overlap-based evidence leaves unclear whether Hessian top-subspace stabilization holds across training setups or independently of parameter-update magnitude, because small updates can inflate overlap.

    Source: §1 · p. 2

  • Method

    The paper introduces an instability metric and uses class-gradient decomposition to identify the top subspaces of the Hessian and gradient covariance matrix.

    Source: §1 · p. 2, §3.1 · p. 6

  • Results

    The authors report Hessian top-subspace stabilization in some training settings, while identifying another setting where stabilization fails despite near-perfect step-wise overlap.

    Source: §1 · p. 2, §5 · p. 11

  • Takeaways & Limitations

    Under gradient flow, the change in the identified subspace is bounded by the Hessian’s operator norm restricted to the bulk, linking slow evolution to separation between top and bulk eigenvalues.

    Source: §1 · p. 2, §5 · p. 11

    The cited prior identifier spans a C-dimensional space, creating a dimensionality mismatch that makes characterization of the top-(C−1) Hessian subspace difficult.

    Source: §4.1 · p. 10

Abstract

from arXiv · show

The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-$(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.

1 Introduction

Prior work treats stabilization of the Hessian’s top subspace as important to neural-network training, but it remains unclear whether stabilization is consistent across training setups or independent of parameter-update magnitude. The paper introduces an update-scaled instability metric and connects Hessian and gradient-covariance subspaces to class-gradient structure and Hessian eigenvalue separation.

1 Introduction

  • The paper asks whether Hessian top-subspace stabilization holds across training setups and independently of parameter-update magnitude, since prior overlap measurements can be confounded by small updates.Prior work links top-subspace stability to analyses of momentum-based optimization and learning-rate schedules.Source: §1 · pp. 1–2
  • An instability metric scales subspace change by the inverse parameter-update magnitude and detects stabilization in standard settings, as well as an architectural setting where overlap alone misses its ablation.The paper reports that modifying network architecture can reveal loss of stabilization not observable with step-wise overlap.Source: §1 · p. 2
  • The mini-batch gradient covariance matrix shares the Hessian’s low-rankness and top-subspace stabilization, and their top subspaces align despite the matrices differing.The gradient covariance matrix is described as highly correlated with the Hessian [24].Source: §1 · p. 2
  • The top-(C-1) subspaces of both matrices are described by centered per-class gradients, and under gradient flow their slow evolution is linked to separation between top and bulk Hessian eigenvalues.The paper’s theoretical account concerns the largest eigenvalue in the bulk subspace and the per-class Hessian.Source: §1 · p. 2

2 Does Hessian Top Subspace Really Stabilize?

The section tests Hessian top-subspace stabilization using both overlap and an instability metric that addresses the overlap metric’s sensitivity to learning rate. Across the reported MLP and two-layer CNN results, overlap alone is insufficient: only K=C−1=9 shows consistently low instability, while a further setting exhibits instability despite near-perfect step-wise overlap.

2 Does Hessian Top Subspace Really Stabilize?

  • The overlap metric is sensitive to learning rate because small parameter updates can make consecutive top subspaces appear arbitrarily close.The argument assumes a universal upper bound on the third derivative and invokes the Davis–Kahan sin Θ theorem.Source: §2 · p. 3
  • The instability metric scales subspace change by inverse parameter-update magnitude and is interpreted as scaled Lipschitzness of the subspace projection.The section recommends examining overlap and instability together to distinguish true stability from an apparent high overlap.Source: §2 · pp. 3–4
  • Only K=C−1=9 shows both near-perfect overlap and consistently low instability in the MLP and two-layer CNN; another setting shows near-perfect step-wise overlap despite failure to stabilize.The authors suggest smaller K may miss rotation within the top subspace, while larger K may include unstable directions.Source: §2 · p. 4, Figure 1 · p. 4
  • The experiments use a two-layer MLP on MNIST, a two-layer CNN on CIFAR-10, and a three-layer CNN with batch normalization on CIFAR-10.Figure 1 encodes K values by color and places overlap and stability metrics in separate columns.Source: §2 · p. 4, Figure 1 · p. 4

3 Structure and Stabilization of Gradient Covariance Matrix

The gradient covariance matrix shares the Hessian’s top-(C−1) subspace alignment and stabilization. A between-class decomposition identifies that subspace through class-mean gradient differences, whose evolution is analyzed under gradient flow.

3 Structure and Stabilization of Gradient Covariance Matrix

  • For classification with cross-entropy, the Hessian decomposition’s N(θ) term is the FIM, while prior work cautions that the empirical FIM does not always capture Hessian properties.The relationship between the empirical FIM and gradient covariance helps explain their observed overlap, but leaves open how much Hessian information the covariance contains.Source: §3 · pp. 5–6
  • The gradient covariance matrix has a Hessian-like eigenvalue profile, near-perfect top-(C−1) subspace overlap at K=C−1=9, and strongest subspace stability at K=C−1.The reported profile separates the top-(C−1) outliers from the bulk.Source: §3 · p. 6

3.1 Identifying the Stable Subspace of the Gradient Covariance

  • The between-class covariance is hypothesized to explain the gradient covariance’s top-(C−1) subspace, with class-mean gradient deviations spanning its candidate directions.Because the weighted class deviations sum to zero, C−1 deviations suffice to span the same space.Source: §3.1 · p. 7
  • In stabilizing MLP and 2-layer CNN settings, the row space of G(θ) stays very close to the gradient covariance’s top-(C−1) subspace throughout training.When the class-mean gradient deviations are linearly independent, their span exactly identifies this subspace rather than merely covering it.Source: §3.1 · p. 7

3.2 Dynamic of Identified Subspace

  • If at least C−1 class-mean gradient deviations are linearly independent, their span is (C−1)-dimensional and its evolution can be studied through its projection matrix.The analysis considers continuous training dynamics induced by gradient flow.Source: §3.2 · pp. 7–8
  • Under gradient flow, the identified subspace’s change scales with projected Hessian-gradient products, while projected Hessian norms in the bulk are consistently small after training’s initial period.Theorem 2 relates projected Hessian terms to ε, an upper bound on the Hessian norm projected onto I−P; observed projected-to-full norm ratios remain small after the initial period.Source: §3.2 · p. 8

4 Stabilization of the Hessian Top Subspace

The top-(C−1) subspace of the gradient covariance matrix aligns with the Hessian’s and can be identified by centered class gradients. The paper connects this identifier’s slow evolution to Hessian spectral separation, while noting that the proposed explanation’s conditions fail in a three-layer CNN with batch normalization.

4 Stabilization of the Hessian Top Subspace

  • The gradient-covariance top-(C−1) subspace aligns with the Hessian’s, and span(g1−g, . . . , gC−g) identifies it and changes slowly when the Hessian bulk and per-class Hessians are small.The authors use this slow evolution to explain stabilization of the Hessian top subspace.Source: §4 · pp. 8–9

4.1 Connecting to the Subspace Identifier from [15]

  • The earlier span(Ghat) covers the Hessian’s top-(C−1) subspace but is C-dimensional, making its characterization of that subspace difficult.The authors present it as a starting point for connecting to the centered class-gradient identifier.Source: §4.1 · p. 10
  • The [15] identifier uses probability-weighted class-gradient aggregates, but dropping those weights preserves their directions approximately and connects them to unweighted class means.The authors describe the weighted gradients as difficult to interpret, motivating the simpler class-gradient formulation.Source: §4.1 · p. 10
  • When class gradients are well separated and the bulk is small relative to the covariance, the weighted-gradient span has near-perfect overlap with the identified top subspace.Centering the class gradients removes the weak full-gradient direction, preserving the top-(C−1) outlier subspace.Source: §4.1 · p. 10

4.2 Accounting for the Reason When Top Hessian Subspace Fails to Stabilize

  • In the three-layer CNN with batch normalization, the centered-gradient span reaches only 0.6 overlap, spectral separation disappears, and the span changes faster; the precise failure mechanism remains future work.Both the identifier’s alignment with the Hessian top subspace and its slow evolution break in this setting.Source: §4.2 · p. 11

5 Conclusion

The paper introduces an update-normalized instability metric to detect Hessian top-subspace stabilization and identifies settings where stabilization fails despite high step-wise overlap. It also connects the Hessian and gradient-covariance top subspaces and relates their evolution under gradient flow to the Hessian’s bulk-restricted operator norm.

5 Conclusion

  • The update-normalized instability metric detects Hessian top-subspace stabilization in some settings and identifies failures that step-wise overlap does not reveal.The metric removes the confound of small parameter updates.Source: §5 · p. 11
  • An explicit identifier links the Hessian and gradient-covariance top subspaces, whose change under gradient flow is bounded by the Hessian’s bulk-restricted operator norm.The paper says separation between outlier and bulk eigenvalues is sufficient to explain the observed early stabilization.Source: §5 · p. 11
  • Open questions include why the class-gradient span identifies the Hessian’s top subspace, how stabilization fails, and why the top-(C-1) eigenvalues grow and separate from the bulk.The paper leaves these questions as future directions.Source: §5 · p. 11

A.1 Common experimental protocol

The protocol uses repeated randomly initialized classifier training with plain SGD and reports results across trials. It documents two training-subset and preprocessing configurations, three model architectures, and the GPU hardware used.

A.1 Common experimental protocol

  • Each setting uses constant-rate vanilla SGD without momentum or weight decay and 10-way cross-entropy; four fresh trials are summarized by mean ± std.Each trial uses a fresh model and optimizer, while the training data is held fixed across trials within a notebook.Source: §A.1 · p. 13

A.2 Datasets and preprocessing

  • The protocols specify 10,000 examples from a 60,000-example training split or 5,000 from a 50,000-example split, with full 10,000-example test sets and raw or standardized inputs.Standardization is per input dimension using the training subset’s mean and standard deviation.Source: §A.2 · p. 14

A.3 Model architectures

  • Architectures include a 784→256→128→10 ReLU MLP and two CIFAR CNNs with two or three width-32 convolution/pooling blocks; the 3-layer CNN uses batch normalization.The MLP has no normalization layers or dropout; both CNNs end in a single linear classifier head.Source: §A.3 · p. 14

A.4 Per-experiment hyperparameters

  • Experiments ran on Google Colab L4 and A100 GPUs.Source: §A.4 · p. 15

B Proof of Theorem 1

The proof differentiates a matrix inverse, introduces a defined quantity, and substitutes gradient-flow dynamics before applying a bound.

B Proof of Theorem 1

C Proof of Theorem 2

The proof uses minimum-eigenvalue reasoning and matrix-block norm inequalities, invoking a lemma for positive-semidefinite shifted matrices and bounding the complementary restriction.

C Proof of Theorem 2

  • The proof begins by invoking the definition of the minimum eigenvalue.Source: C Proof of Theorem 2 · p. 16
  • The proof decomposes the norm of (I − P)H into complementary and cross-block terms, then applies analogous bounds to Hc.Source: C Proof of Theorem 2 · p. 16
  • Because H + νI is positive semidefinite, Lemma 1 bounds its cross-block norm by the norm restricted to the complementary subspace.Source: C Proof of Theorem 2 · p. 16
  • The complementary restriction is bounded by ν + ε, and the proof uses Lemma 1 to handle the remaining Hc cross-block term.Source: C Proof of Theorem 2 · p. 17

D Proof of Theorem 3

The proof invokes the linear independence of the class gradients g_c(θ).

D Proof of Theorem 3

E Proof of Theorem 4

The proof relates the span of class gradients to the top-subspace complement and identifies a projection onto the gradients’ orthogonal complement. Its auxiliary lemmas begin with a positive semidefinite matrix and include an identity involving class-gradient covariance terms.

E Proof of Theorem 4

  • For a unit vector q in the span of the class gradients, the proof states q lies in the orthogonal complement of col(Σ_top(θ)).Source: E Proof of Theorem 4 · p. 18
  • I − uu⊤ is identified as the projection onto the orthogonal complement of the span of the class gradients.Source: E Proof of Theorem 4 · p. 18

F Auxiliary Lemmas

  • An auxiliary lemma begins with a symmetric positive semidefinite matrix M and matrices q and P.Source: F Auxiliary Lemmas · p. 18
  • The auxiliary proof states an identity relating class-gradient covariance Σ_c(θ) to centered class-gradient terms.Source: F Auxiliary Lemmas · p. 19

Related papers

Loading 2609.31983v1…