PaperlayerView exact revision on arXiv

Source-linked AI summary

Symmetry & Critical Points

Yossi Arjevani

arXiv:2408.14445v1cs.LGcs.NAmath.NAmath.OCstat.ML

TL;DR

The paper considers how symmetry breaking arises near symmetric critical points of invariant nonconvex functions, in a context motivated by efficient local optimization of neural-network loss landscapes. It constructs symmetry-inheriting arcs and shows that adjacent critical points are generically symmetry breaking.

  • Problem

    The paper addresses how critical-point symmetry behaves in invariant nonconvex functions, motivated by questions about efficient local optimization of neural-network loss landscapes.

    Source: §1 · p. 2

  • Method

    It constructs arcs from a symmetric critical point that inherit some of its symmetries, so critical points hit by an arc retain at least the arc's symmetry.

    Source: §1 · p. 2

  • Results

    Given a symmetric critical point, adjacent critical points are generically symmetry breaking.

    Source: §1 · p. 2

  • Takeaways & Limitations

    When present, symmetry breaking makes powerful methods for studying nonconvex-problem tractability applicable and helps organize complex sets of critical points.

    Source: §1 · p. 2

    A Taylor approximation determining the extremal characteristic may not always exist, so the authors avoid that situation by imposing generic conditions.

    Source: §4.3 · p. 25

Abstract

from arXiv · show

Critical points of an invariant function may or may not be symmetric. We prove, however, that if a symmetric critical point exists, those adjacent to it are generically symmetry breaking. This mathematical mechanism is shown to carry important implications for our ability to efficiently minimize invariant nonconvex functions, in particular those associated with neural networks.

1. Introduction

The paper develops a way to establish symmetry breaking among critical points adjacent to a symmetric critical point of an invariant nonconvex function. This work grew from questions about local optimization of neural-network loss landscapes and is also aimed at mathematical optimization and theoretical physics readers.

1. Introduction

  • Given a symmetric critical point, arcs that inherit some of its symmetries lead to adjacent critical points that are generically symmetry breaking.A critical point reached by such an arc must be at least as symmetric as the arc.Source: §1 · p. 2
  • Strict convexity lets group averaging make an attained minimum symmetric, whereas nonconvex critical points and minima may have trivial isotropy.Isotropy measures symmetry by identifying the subgroup that fixes a point.Source: §1 · p. 2
  • The work grew from a program to understand efficient local minimization of highly nonconvex neural-network loss landscapes and addresses readers in machine learning, optimization, and theoretical physics.The paper also presents preliminary sections introducing relevant concepts.Source: §1 · p. 2

2. Main results: symmetry breaking critical points

The paper uses tangency sets and arcs to relate a symmetric critical point to nearby critical points and their symmetry. It develops generic structural results for several group representations and connects them to neural-network and tensor-decomposition settings.

2. Main results: symmetry breaking critical points

  • Tangency arcs can break a critical point’s symmetry, and critical points connected by such arcs are themselves symmetry breaking.The paper describes this as a process resembling spontaneous symmetry breaking when arcs connect to a symmetric point.Source: Figure 1 · p. 4, Figure 2 · p. 6
  • Without symmetry, a generic smooth function on R^d has 2d tangency arcs, each tangent to a Hessian eigenspace.The paper also notes that higher-order derivatives can further constrain tangency sets.Source: §2 · pp. 3–4
  • Generically, a tangency set consists of finitely many connected one-dimensional manifolds sharing the center as boundary, and nearby critical points may connect to it by arcs.For the example h, eight arc components connect the center to all its critical points.Source: §2 · p. 5
  • For permutation representations, generic invariant functions are finitely determined, so their tangency sets can be characterized through Taylor-polynomial models.The paper describes this approach as yielding results robust under perturbations.Source: §2 · pp. 4–5
  • For the defining representation of S_d, generic invariant functions have at most exponentially many tangency arcs, with admissible isotropy groups conjugate to S_{d−q} × S_q for q = 0, . . . , ⌊d/2⌋.The paper also identifies finitely determined tangency sets in this setting.Source: §2 · p. 6
  • The framework applies to (M(d, d), S_d)-invariant functions and to symmetric solutions in ReLU networks and tensor decomposition, where symmetry-breaking critical points are expected.The paper says predicted admissible isotropy groups align with those observed empirically for two-layer ReLU networks; its broader neural-network optimization implication is framed speculatively.Source: §2 · pp. 7–8

3. Applications

The applications use symmetry breaking to analyze invariant nonconvex optimization, including Hessian structure, higher-order derivatives, minima transitions, and critical-point families. They connect symmetry to neural-network loss landscapes and report skewed Hessian spectra at symmetry-breaking critical points.

3. Applications

  • The section presents applications of symmetry breaking to invariant nonconvex optimization problems.The applications include prior and novel work.Source: §3 · p. 9

3.1. Hessian, Gauss-Newton and group representation theory.

  • For invariant maps under Sd, the Hessian can have d eigenvalues but at most two distinct values; isotypic decomposition also bounds distinct eigenvalues by the sum of representation multiplicities.The Hessian’s equivariance under the critical point’s isotropy group enables this spectral analysis.Source: §3.1 · pp. 9–10

3.2. Symmetric tensor norms.

  • For invariant f, D^k f(c) is a G-map from Sym^(k−1)(Rd) to Rd, allowing representation methods to compute singular values of its unfoldings.This offers an alternative when second-order information is insufficient and the direct method does not apply to higher derivatives.Source: §3.2 · pp. 10–11

3.3. Annihilation of minima and χ-sufficiency of jets.

  • Adding one neuron turns ΔSd spurious minima into saddles, while adding two turns Δ(Sd−1 × S1) spurious minima into saddles, with the loss essentially unchanged.The transitions tie annihilation of minima to symmetry.Source: §3.3 · p. 12
  • χ-sufficient jets let critical points be classified by reducing the problem to finite jets once sufficiency of a fixed degree is established.A k-jet is χ-sufficient when every function with that jet has the same extremal type at c as the jet.Source: §3.3 · p. 12

3.4. Stabilizing inner products and families of critical points.

  • For natural group and target sequences, definable kernels support critical-point families and Puiseux descriptions of Hessian spectra; the critical-set topology stabilizes for sufficiently large d.Small dimensions can still have bifurcating curves of critical points.Source: §3.4 · pp. 13–14

3.5. Deep symmetry breaking.

  • At symmetry-breaking critical points, Hessian spectra are highly skewed for many representations, qualitatively resembling patterns observed across empirical studies of different architectures and datasets.The extent to which these empirical patterns indicate symmetry-breaking critical points is studied separately.Source: §3.5 · p. 14

4. The tangency set

This section develops a geometric framework for tangency sets, using regularity and definability to describe their structure and the behavior of nearby sets. It also uses polynomial tangency arcs and generic conditions on polynomial jets to analyze their germs, while noting that Taylor truncations may not preserve tangency sets or extremal behavior.

4.1. Preliminaries.

  • Local tangency-set analysis usually assumes smoothness, though finite differentiability is allowed with appropriate restrictions; global structure may instead require o-minimal definability.Source: §4.1 · p. 14

Transversality and stratification.

Transversality theory.

O-minimal theory.

  • Definable sets admit Whitney stratifications and a well-behaved dimension theory, while continuous definable maps become definably trivial after a finite partition of the base.This framework is used when finite differentiability alone may not ensure a suitably structured tangency set.Source: §4.1 · p. 14, O-minimal theory · pp. 18, 20

4.2. Regularity and jet transversality.

  • Tangency sets can be expressed as inverse images under a jet extension, but transversality to Zc excludes functions critical at c, motivating a universal-stratification definition of regularity.The standard stratification of Zc has a singular stratum of codimension 2d+1, which rules out critical points at the origin under the direct transversality condition.Source: §4.2 · pp. 20–22
  • 0-regular functions are dense and open, and sufficiently close functions have isotopic tangency sets when the reference function is 0-regular.The universal set u^-1(Zc) used to define regularity is a closed algebraic set of codimension d−1.Source: §4.2 · pp. 21–22
  • A regular tangency set is either a one-dimensional manifold or a finite union of connected one-dimensional manifolds sharing the origin as a boundary.The two cases correspond to the specified stratum having codimension d − 1 or d, respectively.Source: §4.2 · p. 22
  • If a tangency-set germ is r-determined, its tangency set is isotopic to that of the degree-r Taylor polynomial.Finite polynomial models can simplify analysis, but truncation requires caution: an example has a quadratic Taylor model whose tangency set is not isotopic to the original polynomial’s tangency set.Source: §4.2 · p. 23

4.3. Polynomial tangency sets and the curve selection lemma.

  • A tangency arc’s initial direction is a unit eigenvector of the Hessian, and the direction often, but not always, determines the arc uniquely.Under Proposition 4.27’s stated hypotheses on the cubic term and an invertibility condition, there is a unique real-analytic tangency arc with the specified initial data; the proof uses the real analytic implicit function theorem.Source: §4.3 · pp. 24–25
  • Tangency arcs can relate a function’s extremal behavior to that of its quadratic Taylor approximation, but zero Hessian eigenvalues may require higher derivatives and no determining Taylor approximation need exist.The section avoids cases without such an approximation by imposing generic conditions on jets whose degree depends on the application and function class.Source: §4.3 · pp. 25–26

4.4. Genericity, tensor eigenvalues and constructible sets.

  • For symmetric tensors, the real eigenvalues are finite in number, with their count bounded by ((k − 1)d − 1)/(k − 2).The passage identifies these eigenvalues with critical values of the associated homogeneous polynomial on the sphere.Source: §4.4 · p. 27
  • Property (R1,2) is open in any polynomial subspace where it is nonempty, and it is dense there; for cubic polynomials it is dense and open.The property requires regular eigenvalues of the third derivative restricted to eigenspaces of the Hessian.Source: §4.4 · pp. 26–27

1) (Finite determinacy)

At a critical point, 0-regularity is determined by the third Taylor polynomial; for definable functions, tangency arcs encode local extrema and extremal type.

1) (Finite determinacy)

  • 0-regular germs are 3-determined, and generically the tangency set has 2d arcs approaching the origin along Hessian eigenspaces.The extremal type at the origin is determined by that of the third Taylor polynomial.Source: 1) (Finite determinacy) · p. 28

4.5. Tangency sets definable in o-minimal structures.

  • Near a critical point, the tangency set has finitely many definable components, and its intersection with small spheres stabilizes topologically.Nearby critical points lie in the same path-connected component of the tangency set as the center.Source: §4.5 · pp. 28–29
  • Locally, the tangency set decomposes into finitely many products of a radial interval and connected manifolds, with f constant on each spherical slice and monotone in radius.These properties are stated for sufficiently small radii.Source: §4.5 · p. 28
  • For definable functions, tangency arcs parameterized by arc length attain the minimum and maximum over each small sphere.Non-definable examples need not have an arc attaining the minimum.Source: §4.5 · p. 29
  • For definable functions, tangency arcs determine the extremal type; in polynomially bounded structures, f(γ(t)) = at^α + o(t^α), with a ≠ 0.The asymptotic form follows from the growth dichotomy.Source: §4.5 · pp. 29–30

5. The symmetric tangency set

The symmetric framework defines regularity through equivariant jet transversality and uses representation theory and character calculations to analyze invariant functions and their jets.

5.1. Preliminaries.

  • For invariant functions, the gradient is equivariant and restricts to fixed-point spaces; the Hessian at a critical point is a self-map under its isotropy group.These properties connect critical-point behavior to the group action.Source: Representations · p. 30

Irreducible representations of the symmetric group.

  • The defining permutation representation of S_d splits into the trivial representation and the standard representation on vectors whose coordinates sum to zero.Irreducible representations of S_d are indexed by partitions of d.Source: Irreducible representations of the… · p. 31

Invariant theory.

  • Invariant polynomials form a finitely generated algebra; for permutation representations, generators can be chosen with degree bounded by max{dim V, dim V}.The bound is attributed to.Source: Invariant theory · p. 32

5.2. Regularity and equivariant jet transversality.

  • Equivariant jet transversality extends 0-regularity to invariant functions, with openness, density, and isotopy results; regular tangency sets of permutation representations are finitely determined.The regularity definition uses a universal stratification and equivariant transversality.Source: §5.2 · p. 33

5.3. Stabilizing characters of tensor powers of (Rd, Sd).

  • Character polynomials and weight bounds provide a method for decomposing tensor and symmetric powers into irreducible representations as dimension grows.For a fixed partition α, the character weight is at most |α|; coefficients stabilize once the corresponding partitions are valid.Source: §5.3 · pp. 34–35

Homogeneous invariants of (Rd, Sd).

The section computes invariant spaces and classifies generic tangency sets for permutation representations, then applies representation decompositions to matrix spaces.

Homogeneous invariants of (Rd, Sd).

  • For sufficiently large d, dim P^2(R^d) = 2 and dim P^3(R^d) = 3; the corresponding dimensions for (R^d)^⊗2 are 11 and 52.The latter values are dimensions of quadratic and cubic invariant spaces, respectively.Source: Homogeneous invariants of (Rd, Sd) · pp. 35–36

5.4. Tangency sets of (Rd, Sd)-invariant functions.

  • Generically, the number of (R^d, S_d)-tangency arcs is at most exponential in d, and admissible isotropy groups are conjugates of S_{d−q} × S_q for 0 ≤ q ≤ ⌊d/2⌋.Source: §5.4 · p. 37

5.5. Tangency sets of (M(d, d), Sd)-invariant functions.

  • The matrix representation decomposes into trivial, standard, exterior-square, and V(d−2,2) components, which the analysis uses to study tangency arcs by their leading isotypic component.The diagonal fixed-point space accounts for the occurrence of diagonal-action spurious minima studied in [16].Source: §5.5 · pp. 38–39
  • Cubic invariants vanish on V(d−2,1,1), so constructing analytic tangency arcs there requires the fourth-order homogeneous component.There is no copy of the trivial representation in that component.Source: §5.5 · p. 40
Loading 2408.14445v1…