PaperlayerView exact revision on arXiv

Source-linked AI summary

Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Yossi Arjevani

arXiv:2610.09919v1cs.LGmath.OCstat.ML

TL;DR

Large Hessian kernels and spectral clusters remain unexplained by rank-deficiency accounts in regimes where their assumptions do not apply. The paper analyzes smaller parameter spaces and their embeddings; for N3, the observed behavior persists as width increases, with Adam converging to embedded copies of T*.

  • Problem

    Existing Gauss–Newton rank-deficiency accounts do not explain large Hessian kernels when samples exceed parameters or expected loss is used, nor when the Hessian’s second Gauss–Newton term matters.

    Source: §1.2 · pp. 8–9

  • Method

    The paper analyzes small ambient parameter spaces that realize Bayes-optimal predictors, then embeds them in larger networks, where extra degrees of freedom account for flat directions.

    Source: §1.4 · p. 19

  • Results

    For N3, the observed behavior persists as width increases, with Adam converging to embedded copies of T*; stable occurrence of some symmetry types is not universal.

    Source: §1.4 · pp. 19, 22

  • Takeaways & Limitations

    Adam’s consistent convergence to T* motivates focusing on these minima, whose transverse curvature is substantially smaller.

    Source: §1.4 · p. 18

    Invariance of the objective under Γ alone gives no general lower bound on the stabilizers of critical points.

    Source: §1.4 · p. 20

Abstract

from arXiv · show

Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated outliers. This paper shows that a natural account of these spectral phenomena emerges when the original setting is understood as a departure from a nearby, otherwise hidden, highly symmetric reference. Modifications, including changes to the architecture, data distribution, or parameter metric, expose a nearby reference configuration whose Hessian exhibits rich invariances-ones not accounted for by weight symmetries. There, symmetry enables a precise description of the spectra, forcing high-dimensional kernels and eigenvalues of large multiplicity. Returning to the original configuration breaks the Hessian symmetry and thereby produces the observed hierarchy of clusters and outliers. The framework is developed in some generality, with a detailed analysis of three-layer ReLU networks and applications to convolutional, graph, and transformer models, as well as to the NTK. The same mechanism is further shown to yield analogous spectral structures in layerwise Hessians and the Gauss-Newton matrix.

1 Introduction

The introduction frames Hessian cluster-and-outlier spectra through symmetry-breaking from nearby symmetric reference configurations, whose richer invariances constrain spectra, kernels, and related curvature forms. It outlines the framework’s analytical considerations and extensions, including data alignment, ReLU singularities, Gaussian-mixture settings, optimization-selected critical points, tangency geometry, and the NTK.

1.1 Phenomenology

  • Trained Hessians show a large near-zero bulk and separated outliers across dimensions and settings, with outlier counts scaling with labels in Gaussian mixtures and with input dimension in regression.The pattern also persists with squared loss and increased Gaussian variance; some regression clusters lie close to zero but remain positive.Source: §1.1 · pp. 5–6, Figure 2 · p. 6, Figure 3 · p. 7

1.2 Previous attempts to identify an underlying mechanism

  • Random-matrix accounts can reproduce broad bulk-and-outlier features but do not explain finer fixed-number clusters, spectra in small networks, or what determines structure at a given critical point.The same spectral organization occurs at finite dimensions, beyond the high-dimensional or asymptotic regimes of typical random-matrix models.Source: §1.2 · pp. 6, 8
  • Gauss–Newton rank deficiency cannot explain large near-zero Hessian kernels when samples exceed parameters or for expected loss, and it misses cases where second-derivative terms are significant.The kernel persists in these regimes, including at nonglobal local minima and on large sets of noncritical points.Source: §1.2 · p. 8
  • Layerwise spectra and regression, where one output can still yield Θ(d) outliers, challenge explanations tied to output dimension and motivate a common mechanism across curvature matrices.The introduction also notes related structures in other curvature matrices, including the NTK.Source: §1.2 · pp. 8–9

1.3 Overview: origins of symmetry

  • Hessian invariance is intrinsic, but its spectrum depends on the parameter-space inner product, and this metric dependence can itself generate spectral clusters.The framework calls this mechanism spectral refraction.Source: §1.3 · p. 9
  • At suitable reference configurations, orthogonal Hessian invariances force only a few eigenvalues, some with large multiplicity; for N 3,∅, D4 × C2 forces three nonzero eigenvalue pairs that split when the final-layer bias is restored.Architectural restrictions construct these symmetric references, and the resulting splitting explains the outlier pattern.Source: §1.1 · p. 6, §1.3 · p. 10
  • At generic points, standard weight symmetries can be trivial while Hessian invariances remain rich, imposing spectral constraints and exact layerwise eigenvalue multiplicities.The paper establishes rich invariance structures for restricted Hessians and determines their exact multiplicities.Source: The failure of weight symmetries · p. 10

The failure of weight symmetries.

  • Even modulo global objective symmetries, global minimizers can remain positive-dimensional, yet the Hessian invariances and relevant spectral phenomena persist.The authors argue that weight symmetries and exact identifiability provide too fine-grained a view of the loss landscape.Source: The failure of weight symmetries · p. 11
  • ReLU activation boundaries can add singular terms to the generalized Hessian, so sample-averaged Hessians need not consistently estimate the classical Hessian of expected loss.Calculations involving singular terms were validated using finite differences and Monte Carlo estimates of surface integrals obtained with mollifiers.Source: The failure of weight symmetries · p. 11, analysis · p. 11

analysis.

  • The framework applies directly to Gauss–Newton and Fisher information forms, whose spectra share the Hessian’s qualitative organization, and introduces structural symmetries for NTK and Gram forms.The NTK and Gram-form symmetry action transforms the data, parameters, and tangent perturbations.Source: analysis · pp. 12, 16
  • Combining structural symmetries of the network differential with distributional symmetries of the data yields composite Hessian invariances even when weight symmetries are generically trivial.The same approach can use symmetries of hidden feature–label laws with compatible downstream structural symmetries.Source: analysis · p. 13
  • Initialization-law symmetry forces a high-multiplicity eigenvalue whose magnitude is bounded near zero, yielding a near-zero bulk in random Hessian spectra with high probability.The stated multiplicity is at least (h − 1)^2.Source: analysis · p. 13
  • Large Hessian kernels arise through distinct mechanisms: symmetry can force them across open regions, while well specification can make ker dW N 3 part of the Hessian kernel.In the latter case, this inclusion accounts for roughly half of the spectrum.Source: analysis · p. 14
  • Isotypic decomposition exposes Hessian-invariant subspaces and repeated eigenvalues, while alignment with the data helps estimate their eigenvalue scales.In one example, the quadratic form on an invariant subspace yields the smaller nonzero doublet.Source: analysis · p. 15
  • For the NTK, averaging and deterministic-width limits preserve a composite invariance that refines the isotypic decomposition; the two-Gaussian analysis gives Θ(j−2) positive-eigenvalue decay and a specified multiplicity structure.The additional C2 input factor strengthens the spectral constraints on the initialization-averaged and limiting NTK operators.Source: analysis · p. 17

1.4 Overview: symmetry breaking

  • At symmetric references, structural and data symmetries create Hessian invariances, kernels, multiplicities, and eigenvalue scales; local degeneracy splitting and lower-symmetry optimization-selected points explain richer spectra.The framework distinguishes controlled local splitting from nonlocal symmetry breaking, which can produce a more refined spectrum.Source: §1.4 · pp. 17–18
  • For two-Gaussian minima, D4 × C2 invariance forces nonzero eigenvalues to have multiplicity two; a full final-layer bias breaks it, splitting some eigenvalues under Cauchy interlacing.At random initialization, the averaged Hessian’s invariance also explains the macroscopic near-zero bulk; Adam empirically converges to T⋆, whose curvature transverse to the kernel is smaller.Source: §1.4 · p. 18, Table 3 · p. 19
  • For k-Gaussian mixtures, an orthogonal-Procrustes reference reveals Θ(k2)-dimensional weakly data-aligned subspaces with small eigenvalue scales, whose estimates transfer to the original setting.The setup uses a bias-free network with d = k and h = 2k, and an explicit realization of the Bayes-optimal predictor.Source: §1.4 · pp. 19–20
  • Under Learning Problem 2, the objective has composite invariance S2h × Sd, yet a global minimizer can break this symmetry while retaining the diagonal copy of Sd.A critical point is termed symmetry-breaking when its isotropy group is a proper subgroup of the objective’s invariance group.Source: §1.4 · p. 20
  • Goursat’s lemma and the O’Nan–Scott theorem organize possible subgroup types, while subgroup lattices help classify lower-symmetry critical points without implying that invariance alone bounds their stabilizers.Some symmetry types arise at minima with dead neurons.Source: §1.4 · pp. 20–21
  • Tangency arcs can retain substantial reference symmetry, preserve index, and contain critical points; iterating the construction reveals broader landscape structure, including in symmetric tensor decomposition.For o-minimally definable functions, the arcs can be constructed definably and determine whether a critical point is a minimum, maximum, or saddle.Source: §1.4 · pp. 21–22
  • Tangency sets are computed at the orbit representatives cM, cs, and cm; their isotropy groups are full point stabilizers or maximal proper isotropy subgroups.At cM, eight tangency arcs pass through critical points away from the origin, and iterating the analysis reveals index-zero arcs connecting to cm.Source: §1.4 · p. 23
  • A trained network illustrates observed symmetry breaking from S7 × S2 to approximately ∆(S4 × S1) × S2.The figure also shows a point on a tangency arc from W⋆ with approximate isotropy ∆(C3 × S2).Source: Figure 7 · p. 24, §1.4 · p. 24
  • The figure illustrates forced symmetry breaking in the fixed-point space of D5 ≤ AGL(1, 5) ≤ S5, where AGL(1, 5) is 2-transitive.Although D5 is not maximal in S5, it is a maximal isotropy group for the diagonal action on M(5, 5); similar phenomena occur on parameter manifolds with spherical constraints.Source: §1.4 · p. 24

2 Deep Symmetry Breaking

The section previews symmetry breaking across deeper architectures and models, then analyzes AlexNet in detail. In AlexNet, restoring convolutional directions restricts a permutation action and produces resolutive symmetry breaking.

2 Deep Symmetry Breaking

  • The framework is previewed for deep fully connected networks, AlexNet, GNNs, transformers, unconstrained features, and neural registration, with most applications treated briefly.The AlexNet analysis is more detailed; the other applications are brief previews.Source: §2 · p. 24
  • In deep fully connected networks, the construction of Hessian invariances extends to yield sharp asymptotic spectral estimates as width and depth vary.The relevant permutations act on the full parameter space; in AlexNet, restoring convolutional directions restricts that action on the quotient to D4.Source: §2.1 · p. 25

2.1 AlexNet

  • The AlexNet variant uses a single channel per convolutional layer, no normalization, dropout, or biases, and a 3 × 3 first-layer kernel with unit stride and zero padding.These choices keep the analysis transparent and allow the delta kernel to realize the target; with one channel, no downsampling is essential.Source: §2.1 · p. 25
  • The AlexNet network applies layerwise maps and nonlinearities before a final linear readout, with arbitrary dimensions for the remaining layers.The model uses vectorized input and convolutional kernels for convolutional layers; its tail map is defined as A_L:2.Source: §2.1 · pp. 25–26

alexnet =

The AlexNet analysis identifies quotient-space Hessian invariances and their spectral consequences, including persistent eigenvalue multiplicities and asymptotic scales. The section also sketches an Erdős–Rényi GNN setting and its quotient-spectrum analysis.

alexnet =

  • Structural and distributional actions induce orthogonal quotient actions that preserve the Hessian and Gauss–Newton forms, including a D4 action on the full quotient.The restricted action preserves the forms on the subspace represented by (0, u⊤); the composite actions descend to the quotient.Source: alexnet = · p. 27
  • The standard eigenvalue λstd retains multiplicity at least n − 9 after restoring the first convolutional kernel, because the quotient grows by only eight dimensions.Cauchy interlacing supplies the bound; the pinned eigenspace is identified explicitly.Source: alexnet = · p. 27
  • For zero-padded 3 × 3 convolution with d ≥ 3, the first-layer-coupled subspace has dimension nine and contains 1_n.The mixed term vanishes on its orthogonal complement, which is also orthogonal to 1_n.Source: alexnet = · p. 28
  • D4 symmetry alone permits order d2 distinct eigenvalues, while the resolutive structure permits at most fourteen, or at most thirteen when d = 3.For d = 3, the multiplicity n − 9 is absent.Source: alexnet = · p. 28
  • On the seventeen-dimensional summand, the quotient Hessian has asymptotic spectral structure given by comparison with diag(n2, nI8, I8), with multiplicities resolved by D4 components.The min–max principle yields the tabled structure.Source: alexnet = · pp. 29–30
  • A positive-definite transported parameter metric does not guarantee D4 invariance, so the D4-isotypic components need not be preserved by the ordinary Hessian.Uniform upper and lower bounds on the nonzero singular values of the tail differential suffice to retain the quotient’s asymptotic spectral orders in parameter space.Source: alexnet = · p. 30
  • For the AlexNet variant, the ordinary Euclidean Hessian has kernel dimension p − n − 8 and n + 8 positive eigenvalues.The stated result applies at the specified global minima.Source: alexnet = · pp. 26, 30

2.2 GNNs on Erdős–Rényi graphs

  • The Erdős–Rényi GNN sketch uses a thin model with graph size, feature dimension, and hidden width equal, and states that the second differential equals the Gauss–Newton form at its target.The accompanying figure describes distinct Hessian eigenvalue branches on the parameter quotient at a global minimum.Source: §2.2 · pp. 30, 32, Figure 9 · p. 31

Z (P), sGNN

Across graph, transformer, unconstrained-feature, and registration settings, the section identifies composite Hessian or Gauss–Newton invariances and relates them to spectral multiplicities. It also asks what those multiplicities can reveal about the symmetries behind them.

Z (P), sGNN

  • The composite S_n action in the sGNN induces a diagonal permutation action on effective coordinates; quotienting by the descended Gauss–Newton kernel gives a Hessian quotient with symmetry-forced multiplicities, and the analysis also applies to balanced stochastic block models.For even n, the latter uses the wreath product S_{n/2} ≀ S_2.Source: Z (P), sGNN · p. 32

2.3 Transformers: In-context learning

  • A single-head causal Transformer block exactly realizes the in-context affine task, and its reference point has a C_2 × O(h − 4) Hessian-invariance subgroup with a calculated multiplicity structure.The expected squared loss vanishes there, so the Hessian equals the Gauss–Newton form.Source: §2.3 · pp. 32–33

2.4 Neural collapse, UFM, and spectral refraction

  • At every global minimizer of the unconstrained features model, the Hessian equals the expected Gauss–Newton form, whose kernel dimension h(k + m) − km counts independent flat directions along the minimizer manifold.The global minimizers satisfy WF = Y.Source: §2.4 · p. 34
  • Full wreath-group composite invariance is universal across global minima, while an orthogonal realization of the full group occurs precisely with NC1, NC2, and equal class-mean norms, yielding symmetry-based multiplicities.At the NC1 level, within-class components split; adding NC2 and equal norms extends the orthogonal realization to the between-class factor.Source: §2.4 · p. 35

2.5 Neural registration: ResNet-18

  • Neural registration combines symmetries in the joint hidden-feature–label law with structural symmetries of a network suffix to obtain composite invariances that help explain the original spectrum.For the MNIST ResNet-18, registration strengthens as the NC1 diagnostic decreases toward the final blocks; in a five-layer global minimizer, registration improves while NC1 remains high.Source: §2.5 · pp. 35–36

2.6 Can One Hear the Symmetry of a Critical Point?

  • The inverse question is what eigenvalue multiplicities reveal about their forcing symmetries, but its analysis requires specifying an ambient group and excluding coincidences beyond those forced by symmetry.Source: §2.6 · p. 37

3 Concepts and Methods

The section formalizes architectural, structural, and distributional symmetries and develops how Hessian invariance, symmetries, and flat directions constrain spectra and kernels. It also treats generalized derivatives, higher-order invariances, Gauss–Newton regularity under well-specification, and constraints on averaged and random Hessian spectra.

3.1 Architectural, structural, and distributional symmetries

  • A structural symmetry compatible with the data distribution induces an objective invariance, while an internal structural symmetry does so independently of the distribution.The framework defines architectural, structural, and distributional symmetries through automorphisms of the joint space and their actions on models, losses, and data distributions.Source: §3.1 · p. 39

3.2 The failure of weight symmetries

  • For an orthogonal group action, Hessian invariance makes each isotypic component invariant and forces eigenvalue multiplicities to be multiples of the dimensions of its irreducible representations.The kernel is also group-invariant.Source: §3.2 · p. 41
  • At generic W, the composite transformation cr does not fix W, yet the Hessian retains D4 × C2 invariance; this mismatch also occurs at global minimizers and across the settings studied.The point-stabilizer approach therefore misses relevant Hessian invariances.Source: §3.2 · p. 42
  • Spectral refraction occurs when a nonorthogonal representation preserves the second differential but does not force the Euclidean Hessian to exhibit the corresponding eigenvalue multiplicities.The invariance group of the second differential is independent of the inner product, while the spectrum depends on the inner product used to represent it as a self-adjoint operator.Source: §3.2 · pp. 41–42

3.3 Flat directions and symmetry

  • At critical points, orbit-flat directions lie in the Hessian kernel; at local extrema, all flat directions, including transverse ones, lie in the kernel.At critical points, the orbit-flat directions span a subspace of dimension dim Γ − dim Γp.Source: §3.3 · p. 43
  • For three-layer networks, the Gauss–Newton kernel contains structural orbit directions of dimension 2h generically, or 3h + 1 under cross-entropy; transverse flat directions can yield quadratically growing kernel subspaces.At critical points, the corresponding structural-orbit inclusion also holds for the Hessian.Source: §3.3 · p. 44
  • Under cross-entropy, a fully connected two-output final layer guarantees at least h + 1 zero Hessian eigenvalues (h without bias), wherever the Hessian exists, including noncritical points.The symmetry-and-flat-direction argument is independent of the data and architecture.Source: §3.3 · p. 44

3.4 Generalized derivatives and higher-order symmetry groups

  • For ReLU DAGs, generalized Hessians combine a regular term with a Radon-measure singular part on codimension-one activation interfaces; higher-order invariance transfers to classical Hessians almost everywhere, and everywhere when the objective is C2.The higher-order symmetry groups extend the structural and composite invariances to higher differentials.Source: §3.4 · p. 45

3.5 Gauss–Newton regularity and well specification

  • The pointwise and expected Gauss–Newton forms are regular, and at a twice-differentiable well-specified model the Hessian reduces to the Gauss–Newton form.Well-specification requires the conditional expected loss derivative with respect to the model output to vanish almost everywhere in the input distribution.Source: §3.5 · p. 47

3.6 Invariance of the Averaged Hessian and the Spectrum of a Random Hessian

  • Initialization-averaged Hessian invariances constrain eigenvalue multiplicities through isotypic components, while a general random-operator bound controls negative inertia and eigenvalue counts near zero.For H = Q + R under Proposition 26's assumptions, negative inertia is at most codim(U ⊕ B); with probability at least 1 − δ, at least dim U + dim E − ETr(Π_EQΠ_E)/(δT) eigenvalues are at most T.Source: §3.6 · pp. 47–48

4 Spectral Analysis

The section derives Hessian spectral structure by identifying symmetry-induced flat and invariant subspaces that constrain regression, layerwise, and manifold-restricted spectra. It also examines how breaking or extending these invariances relates to eigenvalue splitting in the two-Gaussian case and small eigenvalues in the k-Gaussian reference.

4.1 The world is (quite) flat21

  • Under h = d and k = 1, transverse flat directions yield Ω(h²) Hessian zero eigenvalues at points with W2, b2 > 0, independently of loss and data distribution.Orbit flat directions alone give only Ω(h) zeros; the transverse directions recover the larger kernel growth.Source: §4.1 · pp. 48, 50, Table 13 · p. 50

4.2 Regression

  • At regression global minima, composite Hessian invariances beyond the point stabilizer force an isotypic multiplicity structure, with flat directions contributing a d(d + 2)-dimensional kernel.For dead-neuron minima, the kernel remains d(d + 2)-dimensional and the quotient retains the same symmetry-forced nonzero multiplicity structure.Source: §4.2 · p. 51
  • In the regression quotient, symmetry-preserved Θ(d²)-dimensional components retain exact multiplicities and uniformly bounded eigenvalues, while spectral refraction yields one Θ(d²) outlier and other unbounded eigenvalues of order Θ(d).The refraction is confined to 3d − 1 = O(d) directions.Source: §4.2 · p. 52

4.3 Analysis by layer and on manifolds: spontaneous SB

  • Composite invariances inherited from the regression Hessian restrict to layerwise parameter subspaces and determine their corresponding isotypic decompositions and multiplicity structures.The natural invariant block involving output-weight variations also includes the second-layer bias variations.Source: §4.3 · pp. 52–53, Table 14 · p. 53
  • On a Frobenius sphere, the identity remains a global minimum, and ambient composite invariances restricted to its trace-zero tangent space constrain the spherical Hessian multiplicities.At the identity, observed multiplicities agree with the predicted structure; a spontaneous symmetry-breaking point has spectrum constrained by its stabilizer.Source: §4.3 · p. 53, Figure 12 · p. 54

4.4 Two Gaussians: resolutive SB

  • At W* in T, where relevant classical derivatives exist and the model is well specified, the Hessian and GN matrix are equal and have 8 zero eigenvalues out of 12 total eigenvalues.A two-dimensional invariant subspace orthogonal to the flat directions carries an intact doublet; splitting of the largest eigenvalue of N^3 provides a structural mechanism for the two isolated eigenvalues observed in Sagun et al. (2016).Source: §4.4 · pp. 54–55

4.5 The Spectrum of a Random Hessian

  • The averaged Hessian acts as a scalar on a subspace B of codimension O(h), and at He initialization the number of negative Hessian eigenvalues is at most (d + 5)h almost surely.The bound is asymptotically sharp in its leading term.Source: §4.5 · pp. 56–57

4.6 k Gaussians: distributional SB

  • For k ≥4, the symmetric Gaussian-mixture reference has large Θ(k^2) components with small eigenvalues; zero-row-sum perturbations have eigenvalues O(1/k), versus a general bound of O(1).The mixture admits an S_k symmetry, and the theorem states that the reference Hessian spectrum is given in Figure 1.Source: §4.6 · pp. 57–58

5 Further Concepts and Proofs

Section 5 develops technical tools for analyzing nonsmooth Hessians and their symmetries, then applies them to spectral bounds and global minimizers in two Gaussian classes. Its extensions beyond the two-Gaussian minimizer analysis are deferred for further technical work.

5.1 Generalized derivatives

  • The section develops generalized-derivative tools for nonsmooth models, with local assumptions connecting the generalized Gauss–Newton form to the classical Hessian.The framework restricts node maps to be componentwise locally DC for the stated regularity results; this excludes Heaviside activations while retaining the architectures considered.Source: §5.1 · pp. 59–61

5.2 Presheaves of symmetry groups

  • Symmetry groups acting on open parameter regions organize invariances of the loss and its derivatives as presheaves under restriction.The framework defines these groups through stabilizers under actions on distributions and tangent-space forms.Source: §5.2 · p. 62

5.3 Quantitative analysis via the GN form

  • The Gauss–Newton form gives direct quantitative spectral control, including uniform and O(k^-1) bounds for Learning Problem 1 and corresponding bounds for Learning Problem 2.The section states that sharper estimates are possible but not needed for its purposes.Source: §5.3 · p. 63

5.4 Characterization of global minimizers for two Gaussians

  • For Learning Problem 1 with d=h=k=2, the section characterizes global minimizers up to permutation and positive rescaling.The argument also extends to other settings considered in the work, but those extensions require additional technical details and are deferred to Arjevani (2027a).Source: §5.4 · pp. 65–66

6 Concluding Remarks

The conclusion presents symmetry and symmetry breaking as an organizing principle for Hessian spectra and summarizes the analytical steps used to connect symmetric references to original models. It also discusses extensions to optimization, invariant tensors, and equivariant networks.

6 Concluding Remarks

  • The framework explains Hessian spectral structure by constructing nearby symmetric references, identifying Hessian and Gauss–Newton invariances, and tracking how symmetry breaking transfers their spectral constraints to original models.The paper's concrete applications focus largely on settings studied in Sagun et al. (2016, 2018), while further extensions are being developed separately.Source: §6 · p. 66

6.1 Optimization

  • In three-layer networks with residual connections, the framework interprets gradient concentration through dynamics converging to fixed-point subspaces and evolving predominantly tangentially to them.The isotypic decomposition organizes the remaining directions.Source: §6.1 · p. 67

6.2 Symmetry and spectral structure of invariant tensors

  • Symmetry can reduce tensor eigenpair computations to smaller polynomial systems and restrict homotopy-continuation paths to those compatible with the symmetry.The framework also describes generic eigencounts for invariant tensor spaces.Source: §6.2 · p. 68

6.3 Equivariance

  • The framework applies unchanged to equivariant networks and accommodates restrictions of the parameter space.At the considered global minima, the predictor is equivariant under symmetries of the data distribution.Source: §6.3 · pp. 68–69

Appendix A. Algebra

This appendix distinguishes the loss's bilinear form from the operator representation induced by a parameter-space inner product.

A.1 Bilinear forms

  • The spectrum and eigenvalue multiplicities can vary with the chosen inner product, while the bilinear form, its invariance group, and its radical remain fixed.The radical is the kernel shared by every operator representation of the form.Source: §A.1 · p. 69

A.2 Group theory and permutation groups

This section points to mathematical background for the group actions and stabilizers arising from transformations of networks and data laws.

A.2 Group theory and permutation groups

  • Network and data-law transformations lead to group actions, stabilizers, direct and semidirect products, and permutation groups.The section cites Dummit and Foote (2004) for general algebraic background and Schmidt (1994, Section 1.6) for Goursat’s lemma.Source: §A.2 · p. 69

A.3 Representation theory

Orthogonal Hessian invariance organizes perturbations into isotypic components, constraining invariant Hessian blocks and eigenvalue multiplicities.

A.3 Representation theory

  • Orthogonal group actions on parameter perturbations yield isotypic components that organize invariant Hessian blocks and constrain eigenvalue multiplicities.Source: §A.3 · p. 69
  • Serre (1977) provides a concise introduction to finite group representations.Source: §A.3 · p. 69
  • Fulton and Harris (1991) provide broader representation-theory coverage, including irreducible representations of symmetric groups.Source: §A.3 · p. 69

B.1 Differential geometry and topology

The section uses group-orbit tangent spaces to identify flat directions and constrained Hessians for spherical parameter settings, drawing on differential geometry and topology.

B.1 Differential geometry and topology

  • Smooth group orbits are manifold directions where an invariant objective is constant, and their tangent spaces help identify flat directions.Source: §B.1 · p. 69
  • When parameters lie on a sphere, the analysis uses constrained Hessians; Lee (2018) and Boumal (2023) cover relevant Riemannian and submanifold geometry.Source: §B.1 · p. 69
  • The background includes group-action symmetry breaking and genericity and stability results for tangency arcs (Arjevani, 2024).Source: §B.1 · p. 70

B.2 Real algebraic and tame geometry

Semialgebraic and o-minimal geometry provide finite smooth stratifications used to handle activation interfaces.

B.2 Real algebraic and tame geometry

  • ReLU networks and their activation regions admit semialgebraic descriptions.Source: §B.2 · p. 70
  • For definable networks, an o-minimal structure gives finite stratifications on which the relevant maps are C2.Source: §B.2 · p. 70
  • This geometric framework supports the treatment of activation interfaces.Source: §B.2 · p. 70

B.3 Variational geometry

Variational geometry distinguishes orbit-generated flat directions from transverse directions by considering tangent cones at level-set points.

B.3 Variational geometry

  • A level-set tangent cone can contain directions not generated by a group orbit.Source: §B.3 · p. 70
  • Such a tangent cone need not be a linear space, providing the geometric language for distinguishing orbit and transverse flat directions.Source: §B.3 · p. 70
  • Rockafellar and Wets (1998) and Clarke et al. (1998) provide background on tangent and normal constructions in variational and nonsmooth analysis.Source: §B.3 · p. 70

B.4 Geometric measure theory

The appendix outlines the mathematical tools used to analyze network Hessians and spectral multiplicities.

B.4 Geometric measure theory

  • Hausdorff measure, the coarea formula, and BV jump derivatives describe activation-interface contributions to Hessian formulas.For locally DC definable networks, the singular part of the second distributional derivative is carried by lower-dimensional activation interfaces.Source: §B.4 · p. 70

B.5 Algebraic geometry, schemes, and intersection theory

  • Polynomial eigenpair equations and intersection theory support counts of invariant-tensor solutions and analyses of tangency arcs.Tangency questions are reduced to finite jets and complexification before genericity is formulated through polynomial systems.Source: §B.5 · p. 70

Appendix C. Analysis

The analysis treats objectives as expected losses under specified data laws, including empirical laws.

Appendix C.1 Measure theory

  • Expected losses are integrated over data laws, with product measures, pushforwards, and Fubini’s theorem also supporting averaging over initialization.For squared loss and cross-entropy, Bayes predictors are conditional means and conditional class probabilities, respectively.Source: §C.1 · p. 70

C.2 Functional analysis and distribution theory

Distributional differentiation, matrix spectral tools, and statistical learning concepts support the paper’s Hessian analysis.

C.2 Functional analysis and distribution theory

  • Treating pointwise loss as a distribution permits parameter differentiation where its classical Hessian does not exist, even when expected loss has a classical Hessian.This formulation permits differentiation under expectation and supports direct analysis of invariances.Source: §C.2 · p. 70

C.3 Matrix spectral theory

  • The min–max principle, Cauchy interlacing, inertia bounds, and eigenvalue perturbation compare spectra and describe how large multiplicities may split into clusters.Source: §C.3 · p. 71

Appendix D. Statistical Learning

  • For squared loss and cross-entropy, Bayes predictors are conditional means and conditional class probabilities, explaining the well-specification condition for identifying Hessians with Gauss–Newton matrices.Source: Appendix D · p. 71
Loading 2610.09919v1…