PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stability

Zhouzi Li, Zixuan Wang, Jian Li

arXiv:2207.12678v2cs.LG

TL;DR

The paper addresses the unresolved explanation of sharpness dynamics in EOS and non-monotonic loss behavior. It analyzes the trajectory in four phases, using output-layer norm as an empirical indicator and proving results under assumptions for two-layer fully connected linear networks. Its analysis links non-monotonic loss to growth in the leading residual direction while the remaining component decreases.

  • Problem

    The mathematical explanation for progressive sharpening and EOS remains unresolved, particularly because sharpness can hover near 2/η while loss decreases non-monotonically.

    Source: §1 · p. 2

  • Method

    The paper analyzes recurring four-phase gradient-descent dynamics, uses output-layer norm as an empirical sharpness indicator, and proves sharpness behavior under assumptions for two-layer fully connected linear networks.

    Source: §1.1 · p. 2, §3 · p. 4

  • Results

    The analysis attributes non-monotonic loss behavior mainly to growth of the leading-eigendirection residual component, while the remaining component follows a converging trajectory.

    Source: §3.3.2 · p. 7, §3.4 · p. 8

  • Takeaways & Limitations

    The paper’s phase-based account explains sharpness changes through output-layer norm and residual dynamics, within its stated assumptions and network settings.

    Source: §1.1 · p. 2

    The theory cannot explain high-frequency sharpness oscillations, and extending the Section 4 results to deeper networks with nonlinear activations remains an open problem.

    Source: §5 · pp. 10–11

Abstract

from arXiv · show

Recent findings (e.g., arXiv:2103.00065) demonstrate that modern neural networks trained by full-batch gradient descent typically enter a regime called Edge of Stability (EOS). In this regime, the sharpness, i.e., the maximum Hessian eigenvalue, first increases to the value 2/(step size) (the progressive sharpening phase) and then oscillates around this value (the EOS phase). This paper aims to analyze the GD dynamics and the sharpness along the optimization trajectory. Our analysis naturally divides the GD trajectory into four phases depending on the change of the sharpness. We empirically identify the norm of output layer weight as an interesting indicator of sharpness dynamics. Based on this empirical observation, we attempt to theoretically and empirically explain the dynamics of various key quantities that lead to the change of sharpness in each phase of EOS. Moreover, based on certain assumptions, we provide a theoretical proof of the sharpness behavior in EOS regime in two-layer fully-connected linear neural networks. We also discuss some other empirical findings and the limitation of our theoretical results.

1 Introduction

The paper addresses the unresolved mathematical explanation of progressive sharpening and Edge of Stability (EOS), where sharpness hovers near 2/η despite classical stability expectations. It proposes a four-phase analysis, uses output-layer weight norm as an empirical indicator, and proves sharpness behavior under assumptions for two-layer fully connected linear networks.

1 Introduction

  • A rigorous explanation for progressive sharpening remains open, while EOS and non-monotonic loss decrease challenge traditional expectations about stability.In EOS, sharpness hovers around 2/η even though classical theory says the learning rate should be below 2/λ for loss to decrease; loss may oscillate yet decline over time.Source: §1 · pp. 1–2

1.1 Our Contributions

  • The paper analyzes EOS through a four-phase account of gradient descent and identifies output-layer weight norm as an indicator for explaining sharpness dynamics.It also offers a partly heuristic account of non-monotonic loss decrease and a more rigorous proof for two-layer fully connected linear networks under assumptions.Source: §1.1 · p. 2

1.2 Related work

  • The paper situates its analysis beyond NTK work, which studies settings where the Hessian changes little, and emphasizes sharpness change during training.It also distinguishes its trajectory-wide account from work focused near minimum-loss manifolds and notes that its analysis is tied to neural-network MSE loss.Source: §1.2 · pp. 2–3

2 Preliminaries

The preliminaries set the paper’s binary-classification, MSE setting and define sharpness through a Gram-matrix quantity linked to the Hessian. They also establish notation for the prediction residual and its gradient-descent dynamics.

2 Preliminaries

  • The paper mainly studies binary classification with mean square error loss, defining the prediction vector and residual as F(t) and D(t)=F(t)−Y.The training objective is introduced using the dataset labels and network predictions.Source: §2 · p. 3
  • The paper uses the largest eigenvalue of the Gram matrix as sharpness, noting it is almost the largest Hessian eigenvalue.The Gram matrix and Fisher information matrix share their nonzero eigenvalues, linking this definition to the Hessian discussion.Source: §2 · p. 3
  • The preliminaries define time-indexed quantities including parameters, sharpness, Gram matrix, predictions, residual, and loss, then introduce residual gradient-flow dynamics.The displayed dynamics passage is introduced as the evolution rule for D(t).Source: §2 · p. 3

3 A Four-phase Analysis of GD Dynamics

The section divides EOS training into recurring phases and uses output-layer norm as an empirical proxy for sharpness changes. Under simplifying assumptions, it explains phase-specific dynamics and attributes non-monotonic loss behavior chiefly to the leading-eigendirection component of the residual.

3.3 Detailed Analysis of Each Phase

  • The phase analysis is explicitly approximate: it adopts simplifying assumptions and first-order dynamics, including gradient flow below 2/η.The assumptions include a bound on sharpness when training does not diverge and increasingly weaker conditions on eigendirection movement.Source: §3 · p. 4, §3.3.1 · pp. 5–6

3.1 A Four-phase Division

  • The four recurring phases track sharpness relative to 2/η: it rises below the threshold, overshoots, falls while loss may rise, then drops below threshold as loss recovers.The trajectory returns to Phase I after the loss falls toward its level before Phase III.Source: §3.1 · p. 4

3.2 The Norm of the Output Layer Weight

  • Output-layer norm and sharpness usually change in the same direction, making the norm an empirical indicator despite occasional mismatches.The paper reports this pattern across two-layer linear, deep fully connected, and ResNet18 experiments; Figure 1 also depicts a rapid sharpness drop after crossing 2/η.Source: §3.2 · pp. 4–5, Figure 1 · p. 5

3.3.2 Detailed Analysis

  • For Phase I, small initial output and spectral assumptions yield increasing output-layer norm and sharpness until sharpness crosses 2/η or loss reaches zero.The authors also state that the condition used to establish the residual-output inner-product sign can be relaxed.Source: §3.3.2 · p. 7
  • In Phase II, the leading residual component grows geometrically; once the residual norm exceeds the label norm, the output-layer norm and sharpness decrease.The Phase II argument requires slowly changing principal directions and a non-negligible initial component along the leading eigenvector.Source: §3.3.1 · p. 6, §3.3.2 · p. 7
  • In Phase III, sharpness remains above 2/η while decreasing, and the loss may rise; after crossing below threshold, Phase IV dynamics decrease the loss and eventually restart Phase I.The authors state that sharpness provably drops below 2/η in the two-layer linear setting; below threshold, the leading residual component decreases geometrically.Source: §3.3.2 · p. 7

3.4 Explaining Non-monotonic Loss Decrement

  • The loss’s non-monotonicity is mainly attributed to growth of the leading-eigendirection residual component, while the remaining component tracks a convergent sequence and decreases.The remainder is shown to be close to an auxiliary trajectory under spectral and eigendirection assumptions.Source: §3.4 · pp. 8–9

4 A Theoretical Analysis for 2-Layer Linear NN

The two-layer linear-network analysis proves progressive sharpening under stated assumptions and examines sharpness and parameter dynamics in the EOS regime. It also relates the output-layer norm and Gram-matrix structure to sharpness behavior.

4.1 Setting and basic notations

  • The analysis studies two-layer networks with linear activation and assumes binary labels, rank-r X⊤X, n ≫ r, and existence of a global minimum.It writes the GD dynamics for D(t) using a Gram matrix combined with second-order terms.Source: §4 · p. 9, §4.1 · p. 9

Phase I and Progressive Sharpening:

  • Sharpness increases at every step in the stated progressive-sharpening interval, ending when the loss norm is small or λmax(M*) first exceeds 1/η.The theorem's conclusion is conditioned on assumptions about the data spectrum and other quantities.Source: Phase I and Progressive Sharpening · p. 9
  • The proof approximates the Gram matrix using X⊤X, whose eigenvectors thereby approximately align with those of M(t), and establishes ∥A(t)∥² as an indicator of sharpness.This result is presented as a justification for an assumption used in the broader analysis.Source: Phase I and Progressive Sharpening · p. 10
  • When sharpness rises above 2/η during progressive sharpening, it eventually falls below 2/η; meanwhile, |D(t)⊤v1| grows geometrically under the theorem's conditions.The authors interpret the decline as a regularization effect of gradient descent and the projection growth as occurring in Phases II and III.Source: Phase I and Progressive Sharpening · p. 10
  • Theorem 2 bounds the distance between R(t) and an auxiliary trajectory R′(t), which the authors use to explain their similarity in Phase IV.The theorem's conclusions also include eventual sharpness decline and geometric growth of |D(t)⊤v1|.Source: Phase I and Progressive Sharpening · p. 10

5 Discussions and Open Problems

The discussion identifies limits in the theory's account of sharpness dynamics and lists extensions that remain open. The experimental setup uses a 1,000-sample binary subset of CIFAR-10.

5 Discussions and Open Problems

  • The theory cannot explain sharpness's high-frequency oscillations, and ∥A∥² and sharpness can have different long-run trends despite matching low-frequency trends.The analysis relies on ∥A∥ changing in the same direction as sharpness most of the time.Source: §5 · p. 10
  • A complete account of sharpness dynamics must consider factors beyond ∥A∥, including other layers; weakening assumptions and extending results to deeper nonlinear networks remain open problems.The authors also identify making heuristic arguments more rigorous as an open problem.Source: §5 · pp. 10–11

A.1 Dataset

  • Experiments in Appendices B and D use 1,000 CIFAR-10 samples: the first 500 examples from each of classes 0 and 1, labeled ±1.The dataset size is limited under GPU memory constraints when computing model Gram matrices.Source: §A.1 · p. 13

A.2 Network Architecture

The experiments span a simple linear network, fully connected and convolutional networks, and ResNet18. The CIFAR-10 subset used here is smaller than Cohen et al. [6]’s, but the reported loss and sharpness properties are similar.

A.2 Network Architecture

  • Experiments include a one-hidden-layer linear network, four-hidden-layer fully connected networks, convolutional networks, and ResNet18.The fully connected and convolutional experiments use tanh, ReLU, and ELU activations; the convolutional models have two convolutional and two max-pooling layers.Source: §A.2 · pp. 13–14, Table 1 · p. 13
  • The CIFAR-10 subset is smaller than the 5,000-example subset used by Cohen et al. [6], yet the loss and sharpness properties are reported as similar across datasets.The smaller subset reflects computation limits when calculating the Gram matrix.Source: §A.2 · p. 13

B Further Experiments and Discussions

Across linear, fully connected, and convolutional networks, experiments examine the four-phase EOS trajectory and the relation between sharpness and output-layer A-norm. A-norm tracks sharpness in many settings, though anomalies become more common after early oscillations.

B Further Experiments and Discussions

  • The appendix tests the sharpness–A-norm observation on a 1,000-example CIFAR-10 subset using full-batch gradient descent, with initialization sharpness below 2/η.It includes linear, fully connected, and convolutional networks; ResNet18 is excluded from these appendix experiments because GPU memory limits Gram-matrix calculations.Source: B Further Experiments and Discussions · p. 14

B.1.1 Linear Networks

  • Even a simple linear network enters EOS, and its trajectory shows the four-phase division; sharpness correlates positively with A-norm.Although total loss oscillates considerably, the residual norm decreases more steadily.Source: §B.1.1 · p. 14, Figure 4 · p. 14

B.1.2 Fully-connected networks

  • In three five-layer fully connected networks, sharpness and output-layer A-norm are positively correlated most of the time during progressive sharpening and the first few oscillations.Anomaly points become more frequent after a few oscillations, when sharpness oscillates more frequently around 2/η.Source: §B.1.2 · p. 15

B.1.3 Convolutional Networks

  • Convolutional networks retain the positive sharpness–A-norm correlation and show the four-phase division during the first few sharpness oscillations.The experiments also report the same anomaly seen in fully connected networks.Source: §B.1.3 · p. 16, Figure 8 · p. 16

B.1.4 Gaussian Data on Linear Networks

Gaussian-data experiments show progressive sharpening and EOS in a two-layer linear network, with sharpening degree varying by label choice and learning dynamics. The discussion also identifies limits to A-norm tracking and to the four-phase account in late training.

B.1.4 Gaussian Data on Linear Networks

  • Gaussian data and random ±1 labels still produce progressive sharpening and EOS in a two-layer linear network, but rapid convergence to zero loss makes the EOS behavior atypical.The reported experiment uses a width-200 network and 1,000 standard-Gaussian datapoints.Source: §B.1.4 · p. 17, Figure 11 · p. 17
  • With anisotropic-Gaussian inputs, top-eigenvector-aligned labels yield less progressive sharpening than bottom-eigenvector-aligned labels; random labels produce a degree similar to CIFAR-10.The authors associate faster convergence with a shorter progressive-sharpening phase and less sharpening.Source: §B.1.4 · p. 17, Figure 12 · p. 18

B.2 Further Discussions on the Relation between A-norm and Sharpness

  • A-norm is an imperfect indicator: inner-layer changes may affect sharpness near A-norm trend reversals, and other layers matter for small, frequent sharpness oscillations.In late training, sharpness crosses 2/η more frequently, so the four-phase division no longer strictly applies and those stages remain beyond the analysis.Source: §B.2 · pp. 17–18

C.1 Detailed Settings

  • The detailed linear-network setup defines the output F and residual D, uses MSE loss, and assumes ±1 labels, rank-r X⊤X, and n ≫ r.The input Gram matrix is assumed to have norm Θ(n); GD is stated to travel only in the column space of X⊤X under the rank condition.Source: §C.1 · pp. 18–19

C.2 Phase I and Progressive Sharpening

This section analyzes Phase I progressive sharpening, tracing the increase in sharpness-related quantities alongside changes in the output-layer weight norm and residual norm.

C.2 Phase I and Progressive Sharpening

  • The top eigenvector of M(t) is approximately v1, so the paper analyzes Λ*(t) as an approximation to the actual sharpness Λ(t).Corollary C.1 bounds the squared alignment ⟨v1,u⟩² below by 1−c4.Source: §C.2 · p. 21
  • The proof requires mild over-parameterization, with m = Ω(n²) when λr = Θ(1), but does not necessarily require the NTK regime.The analysis proves progressive sharpening only until λmax(M*(t)) reaches 1/η; extending it to 2/η remains open.Source: §C.2 · p. 22
  • Sharpness dynamics depend on ∥A(t)∥² and −D(t)⊤F(t), whose sign and growth conditions underpin the proof of increasing approximate sharpness.The lemmas establish a sufficient condition for ∥A(t)∥² to grow and show −D(t)⊤F(t) > 0 over a time interval.Source: §C.2 · pp. 21, 28, 29
  • Λ*(t) increases until ∥D(t)∥² ≤ ϵ² + 5ϵλr, establishing progressive sharpening under Assumptions C.1 and C.2.The theorem’s stated range also ends if λmax(M*(t)) exceeds 1/η, before the empirical sharpness threshold 2/η.Source: §C.2 · pp. 21, 22, 29
  • Λ*(t) keeps increasing before the residual norm D(t) decreases to the referenced value.Source: §C.2 · p. 31

C.3 Edge of Stability (Phase II-IV)

The EOS analysis decomposes the residual dynamics into a principal direction and its orthogonal remainder, then uses bounds and growth estimates to show that sharpness above a threshold is temporary. The proof relies on stated assumptions, including control of the perturbation and the residual’s principal component.

C.3 Edge of Stability (Phase II-IV)

  • The bound ∥Γ(t)∥ = O(1/m) is assumed in EOS because the progressive-sharpening proof no longer establishes it after sharpness reaches 2/η.The bound is empirically reported to hold up to a constant factor in EOS, but is made an assumption there.Source: §C.3 · p. 31
  • Under Assumptions C.3–C.5 and λ1(X⊤X) ≥ 2λ2(X⊤X), the proof bounds R(t) from above and establishes that sharpness eventually drops below 2/η.Source: §C.3 · p. 32
  • When sharpness stays above 2/η(1 + ρ) across consecutive iterations, |D(t)⊤v1| grows geometrically under Assumption C.4.The growth factor is stated to be 1 + c7, with c7 > 0 related to c3.Source: §C.3 · pp. 31–32
  • M*(t) is treated as the matrix controlling D(t)’s update and is approximated by a weighted average of M(t) and M(t + 1), plus a small perturbation.This decomposition supports a corresponding decomposition into an X⊤X component and a small noise term.Source: §C.3 · pp. 32–33

C.4 Our Results and the NTK Regime

The paper distinguishes its setting from NTK convergence analyses, which require small learning rates, and reports PS and EOS even with negligible movement relative to initialization. The appendices empirically check assumptions used in Sections 3 and 4, including with a 1/√m scale coefficient for the linear network.

C.4 Our Results and the NTK Regime

  • Even when W(t) changes by several orders of magnitude less than its initialization, PS and EOS still occur with a not-so-small learning rate, unlike the quadratic setting where EOS cannot happen.The authors also note that their current proofs require m = Ω(n²) and λr = Ω(1), and leave tightening these bounds as future work.Source: §C.4 · p. 37

D.1.1 MA has small eigenvalues

  • Across fully connected linear, tanh, and convolutional networks, ∥MA∥ stays near zero and hardly increases during training.Figure 13 reports that ∥MA∥ remains much smaller than sharpness throughout the trajectory.Source: §D.1.1 · p. 37, Figure 13 · p. 38

D.1.2 Assumption 3.7

  • In the binary setting, the largest eigenvalue of M dominates the second, which remains below 1/η in the reported verification of Assumption 3.7.The passage relates this to an outlier separated from the bulk of the spectrum.Source: §D.1.2 · p. 38, Figure 14 · p. 38

D.1.3 Assumption 3.2 (First Order Approximation of GD)

  • The first-order approximation closely matches the actual updates for both D(t) and ∥A(t)∥² in progressive sharpening and remains close most of the time in EOS.Higher-order terms spike occasionally in EOS, while the first-order term spikes more strongly when they do.Source: §D.1.3 · p. 38, Figure 15 · p. 39

D.1.4 Assumption 3.3 (Gradient flow for the PS phase)

  • For progressive sharpening, the appendix reproduces Cohen et al.’s comparison of gradient-flow and gradient-descent trajectories, with the 2/η sharpness crossing marked.The figure plots trajectory distance against iteration and sharpness against iteration × learning rate; the passage refers readers to Cohen et al. [6] for further experiments.Source: §D.1.4 · p. 39, Figure 16 · p. 39

D.1.5 Assumption 3.5 (iii) (principal directions moves slowly)

  • The leading eigenvector changes slowly: its similarity to its initialized direction remains at least 0.98 over long training, although small-eigenvalue eigenvectors may change drastically.The passage also notes that near-minimum alignment of top Hessian eigenvectors was proved by Mulayoff et al..Source: §D.1.5 · p. 39, Figure 17 · p. 40

D.2.1 Assumption 4.1

  • On the 1,000-example CIFAR-10 subset, adjacent-eigenvalue ratios support Assumption 4.1 with χ ≈ 38; subtracting the sample mean reduces the reported ratio to χ ≈ 3.The mean-subtracted data are not full rank, so the zero minimal eigenvalue is excluded from the check.Source: §D.2.1 · p. 40, Figure 19 · p. 41

D.2.2 Assumption 4.3

The appendix reports that the assumed EOS bound on ∥Γ(t)∥ remains small at tested widths, while acknowledging that its spikes are not captured by the analysis. It also distinguishes this bound from small movement of W(t), and reports PS and EOS even when that movement is negligible relative to initialization.

D.2.2 Assumption 4.3

  • ∥Γ(t)∥ is reported at no more than 24/m for m = 40, 80, 160, and 200, and generally decreases as width grows despite EOS spikes.The authors say their results do not reflect the spikes and provide no formal proof that ∥Γ(t)∥ grows by at most a constant factor in EOS.Source: §D.2.2 · pp. 41–42, Figure 20 · p. 41
  • Assumption 4.3 does not imply small movement of W(t): in the tested widths, its movement is significant relative to the initialization norm.Figure 21 compares ∥ΔW(t)∥ with ∥W(t)∥ and indicates substantial movement.Source: §D.2.2 · p. 42, Figure 21 · p. 42

D.2.3 Comparison with the NTK regime: the non-quadratic property

  • Using initialization ten times the standard initialization and widths m = 1,000, 2,000, 4,000, and 8,000, the authors show EOS can happen even when W(t)’s movement is negligible relative to W(0).The norm of W(t) grows larger as m increases in these experiments.Source: §D.2.3 · p. 42

D.3 Experiments in Section 5

Experiments indicate that layers beyond the output layer contribute to sharpness dynamics: freezing more layers slows progressive sharpening and weakens sharpness oscillations. EOS can nevertheless occur while the first-layer weights remain close to initialization.

D.3 Experiments in Section 5

  • Freezing more layers slows progressive sharpening and weakens sharpness oscillations, indicating that layers beyond the output layer influence sharpness.The experiments freeze layers while calculating sharpness from gradients of all parameters; the reported trend supports the assumption that the output-layer norm is positively related to sharpness.Source: §D.3 · pp. 42–43
  • EOS still occurs when ∥W(t)∥/∥W(t) − W(0)∥ > 10 for m = 1000, 2000, 4000, and 8000.The figure caption interprets this as evidence of the intrinsic non-quadratic property of neural networks in the EOS regime.Source: Figure 22 · p. 43
  • Sharpness results are compared across four independent training runs with 0, 1, 2, or 3 outer layers locked, with other hyperparameters held fixed.The figure reports a cooperative effect of all layers on sharpness.Source: Figure 23 · p. 43, Figure 24 · p. 44

E.1 Proofs in Section 3

The appendix supplies proofs for results about progressive sharpening and analyzes how those results can extend beyond fixed Gram-matrix eigenvectors. Its geometric view links the residual trajectory to increasing output-layer norm and sharpness, while identifying eigenvector-change conditions that limit the guarantee.

E.1 Proofs in Section 3

  • In Phase I, D(t)⊤F(t) < 0 under Assumptions 3.5(i) and 3.3, a condition used in the proof of progressive sharpening.The proof begins from the fixed-eigenvector assumption for the Gram matrix.Source: §E.1 · p. 44
  • When Λ(t) = (2 + τ)/η under the stated conditions, D(t)⊤v1(t) grows geometrically, while ∥D(t)∥ > ∥Y∥ implies a positive D(t)⊤F(t).The geometric-growth condition requires τ > 1/(1 − ϵ2 − 1/c) − 1; the positive inner product also implies a decrease in squared output-layer norm under Assumption 3.2.Source: §E.1 · p. 45

E.2 Progressive Sharpening under Weaker Assumptions

  • The proof of progressive sharpening can replace fixed Gram-matrix eigenvectors with a more realistic condition on eigenspace change.Under sufficiently slow eigenvector-direction changes, D(t) remains in H(t) and therefore in B; this implies increasing ∥A∥ and sharpness.Source: §E.2 · p. 47, §E.2.1 · p. 47

E.2.1 A dynamical system view

  • For constant M, D follows a linear system; the geometric view uses D(t)⊤F(t) < 0 to characterize membership in B.With slowly changing eigenvectors, the trajectory remains in H(t) ⊂ B, linking this view to increasing output-layer norm and sharpness.Source: §E.2.1 · p. 47
  • Without an assumption on eigenvector-direction change, the proof cannot guarantee D(t) ∈ B for all t; sufficiently fast changes may take the trajectory outside B.The authors suggest that sharpness may still increase until near 2/η, but leave proving this for future work because gradient flow ceases to approximate gradient descent well near that threshold.Source: §E.2.1 · p. 47

E.2.2 Progressive sharpening when eigendirections change slowly

The appendix extends the progressive-sharpening argument to slowly changing eigendirections, tests its condition empirically, and provides supporting equation proofs and preliminaries. The guarantee remains conditional, and the empirical check does not hold for every eigenvector index.

E.2.2 Progressive sharpening when eigendirections change slowly

  • Under the lemma’s conditions, D(t) remains in H(t) ⊂ B until t1, the first time some D(t)⊤vi(t) becomes nonpositive.The proof does not show that each projection decreases at all times when M(t) changes; instead it establishes containment in the hypercube.Source: §E.2.2 · pp. 48–49

E.2.3 Empirical verification of inequality (20)

  • Inequality (20) holds for i = 1, 5, and 50 during the first 100 training iterations, but can fail for very large i.The authors report that contributions from eigenvectors with very small eigenvalues tend to cancel empirically, leaving the sign of the total sum unaffected.Source: §E.2.3 · p. 49, Figure 26 · p. 49

E.3 Proofs of equations in Appendix C

  • The appendix derives the residual-vector update by expanding changes in the output-layer vector A and the weight matrix W.It also gives the Gram-matrix update in terms including changes in ∥A∥² and WᵀW.Source: §E.3 · pp. 49–50

F Missing Preliminaries

  • The paper defines sharpness as the largest eigenvalue of M, which empirically closely approximates the largest Hessian eigenvalue.M and the Fisher information matrix share the same nonzero eigenvalues; the paper does not assume training is in the NTK regime.Source: F Missing Preliminaries · pp. 52–53

Related papers

Loading 2207.12678v2…