PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, Johanni Brea

arXiv:2105.12221v2cs.LG

TL;DR

The paper asks how permutation symmetries shape critical points and connectivity in overparameterized neural-network landscapes. It maps narrower-network points to equal-loss manifolds, characterizes global-minimum and symmetry-induced critical subspaces, and finds that their prevalence reverses between mild and vast overparameterization.

  • Problem

    The paper studies how permutation symmetries generate critical points and affect connectivity in non-convex overparameterized neural-network loss landscapes.

  • Method

    The authors exploit permutation symmetry and further parameterization invariances to map narrower-network points into manifolds and count affine subspaces combinatorially.

  • Results

    Symmetry-induced critical subspaces dominate global-minimum subspaces under mild overparameterization, whereas global-minimum subspaces dominate when overparameterization is vast.

  • Takeaways & Limitations

    The results link optimization behavior to the width-dependent geometry and relative abundance of global-minimum and symmetry-induced critical subspaces.

  • Takeaways & Limitations

    It remains open whether overparameterized networks contain critical points beyond the symmetry-induced ones and whether all such points are strict saddles.

Abstract

from arXiv · show

We study how permutation symmetries in overparameterized multi-layer neural networks generate `symmetry-induced' critical points. Assuming a network with $ L $ layers of minimal widths $ r_1^*, \ldots, r_{L-1}^* $ reaches a zero-loss minimum at $ r_1^*! \cdots r_{L-1}^*! $ isolated points that are permutations of one another, we show that adding one extra neuron to each layer is sufficient to connect all these previously discrete minima into a single manifold. For a two-layer overparameterized network of width $ r^*+ h =: m $ we explicitly describe the manifold of global minima: it consists of $ T(r^*, m) $ affine subspaces of dimension at least $ h $ that are connected to one another. For a network of width $m$, we identify the number $G(r,m)$ of affine subspaces containing only symmetry-induced critical points that are related to the critical points of a smaller network of width $r<r^*$. Via a combinatorial analysis, we derive closed-form formulas for $ T $ and $ G $ and show that the number of symmetry-induced critical subspaces dominates the number of affine subspaces forming the global minima manifold in the mildly overparameterized regime (small $ h $) and vice versa in the vastly overparameterized regime ($h \gg r^*$). Our results provide new insights into the minimization of the non-convex loss function of overparameterized neural networks.

1. Introduction

The paper studies how permutation symmetries and related invariances shape critical points and connect global minima in overparameterized neural-network loss landscapes. It shows that small overparameterization connects discrete minima, while the balance between global-minimum and symmetry-induced critical subspaces reverses as width increases.

  • Introduction: The study addresses concerns that non-global critical points impede optimization by explicitly analyzing symmetry-induced saddles and their relation to global minima.Prior work reported connected minima and global convergence in highly overparameterized settings, motivating this geometric analysis.
  • Main Contributions: Adding one neuron to every hidden layer connects r_1^*! ··· r_{L−1}*! permutation-equivalent zero-loss minima into one manifold.The result assumes a minimal network reaches a unique zero-loss minimum up to permutations.
  • Main Contributions: For two-layer width m = r^* + h, the connected global-minima manifold contains T(r^*, m) affine subspaces, each of dimension at least h.The characterization includes all zero-loss points under the stated smooth-activation and infinite-data conditions.
  • Main Contributions: When h ≪ r^*, symmetry-induced critical subspaces greatly outnumber global-minimum subspaces, whereas h ≫ r^* reverses this relationship.The paper describes these regimes as a proliferation of saddles versus a large global-minima manifold.
  • Related Work: Adding or removing a single neuron can radically change connectedness without changing loss, highlighting the geometric impact of overparameterization.The paper presents this as a central novelty alongside its width-dependent scaling analysis.

2. Symmetric Losses

Permutation symmetry makes loss landscapes invariant under unit rearrangements and creates symmetry subspaces where gradient flow has constrained dynamics. The paper defines these subspaces, illustrates their geometry, and notes an infinite-time convergence caveat.

  • Symmetric Losses: Permutation-symmetric losses apply across mixture models, matrix factorization, and neural networks, with units permuted without changing the parameterized loss.The framework treats neural networks as the main application while abstracting away model-specific parameterizations.
  • Symmetric Losses: The symmetry-subspace geometry includes permutation-symmetric global minima and saddles, with no gradient component pointing outside the displayed subspace.Figure 2 illustrates this behavior for a two-variable permutation-symmetric loss.
  • Symmetric Losses: A symmetry subspace H_i1,...,ik constrains selected units to identical parameter vectors and has dimension D(m − k + 1).Each equality constraint removes D degrees of freedom, and pairwise-equality subspaces are the largest such subspaces.
  • Symmetric Losses: Gradient flow starting inside a symmetry subspace remains there, while trajectories initialized outside all such subspaces cannot enter one in finite time.Permutation symmetry makes the gradient tangent to the subspace because components toward neighboring regions cancel.
  • Symmetric Losses: The finite-time exclusion does not rule out convergence to a symmetry-subspace critical point at infinite time when it attracts in orthogonal directions.This is the explicit caveat attached to the gradient-flow lemma.

3. Foundations: Invariances in 2-Layer ANNs

The paper identifies permutation and replication-related invariances that create equal-loss affine subspaces in two-layer networks. These subspaces form an expansion manifold whose geometry, dimension, and connectivity can be counted explicitly.

  • Permutation symmetry makes neuron-reordered parameterizations equivalent and leaves the loss invariant under neuron permutations.
  • A reducible neuron can be removed when its outgoing weight is zero or when two incoming weight vectors coincide, preserving the network function.
  • Replicating an irreducible r-neuron point in width m creates affine equal-loss subspaces by splitting outgoing weights among replicated or zero-type neurons.
  • The expansion manifold contains exactly the counted affine subspaces, each with dimension at least min(din, dout)(m − r), and is connected whenever m > r.
  • Connectivity follows by linking neighboring subspaces through three line segments for transpositions, after which arbitrary permutations are reached through compositions of transpositions.

4. Overparameterized ANN Landscapes

In two-layer overparameterized networks, expansion manifolds connect permutation-equivalent global minima and organize symmetry-induced critical subspaces. Their relative counts favor saddles under mild overparameterization but global minima under vast overparameterization.

  • 4.1. The global minima manifold: Adding one neuron to each layer connects all permutation-equivalent zero-loss minima into a single manifold.
  • 4.1. The global minima manifold: The global-minimum manifold is a connected union of T(r*, m) affine subspaces of dimension at least h for width m = r* + h.
  • 4.1. The global minima manifold: Under the stated smoothness and activation conditions, every zero-loss point lies in the expansion manifold, while standard softplus and sigmoid activations require additional invariances.
  • 4.2. Symmetry-induced critical points: A critical point of a narrower network expands into G(r, m) affine critical subspaces, and strict saddles retain an escape direction across these affine transformations.
  • 4.2. Symmetry-induced critical points: The hierarchy labels k-th-level saddles by equivalence to minima of networks of width r* − k, with higher levels usually attaining higher losses.
  • 4.2. Symmetry-induced critical points: For small h, symmetry-induced saddle subspaces outnumber global-minimum subspaces, whereas for h much larger than r*, global-minimum subspaces dominate.

5. Multi-Layer ANNs

The multi-layer construction sequentially expands adjacent weight-matrix pairs, preserving network outputs and extending the two-layer connectivity and counting results. With one extra neuron in every hidden layer, the resulting global-minimum manifold is connected, while saddle dominance depends on width and depth.

  • The expansion manifold generalizes two-layer results by sequentially expanding every adjacent pair of weight matrices in an irreducible multi-layer network.
  • Each expansion preserves the network output and loss, although expansion order changes the final manifold; expanding from the last layer yields the largest manifold.
  • If every hidden layer has at least one extra neuron, the multi-layer expansion of a unique minimal-width global minimum is a connected zero-loss manifold with a counted number of affine subspaces.
  • Applying expansions to all hidden layers preserves criticality and produces affine subspaces of symmetry-induced critical points in the wider network.
  • In mildly overparameterized multi-layer networks, saddle counts dominate increasingly with depth, whereas vast overparameterization strengthens global-minimum dominance.

6. Conclusion & Discussion

The paper characterizes affine-subspace geometry for global minima and symmetry-induced critical points in overparameterized networks. It quantifies how their numbers scale with width while identifying open questions about other critical points and strict-saddle status.

  • 6. Conclusion & Discussion: The paper explicitly characterizes global-minimum and symmetry-induced critical-point geometries as connected or enumerated affine subspaces.It notes that global minima form a manifold of connected affine subspaces, while symmetry-induced critical points form an explicit number of affine subspaces.
  • 6. Conclusion & Discussion: Whether other critical points exist beyond the symmetry-induced ones and whether all such points are strict saddles remain open questions.
  • 6. Conclusion & Discussion: The main result quantifies how the numbers of global-minimum and symmetry-induced critical subspaces scale as network width grows.
  • 6. Conclusion & Discussion: The appendix supplies proofs, combinatorial derivations of T and G, numerical analyses, symmetric-loss examples, and multi-layer generalizations.

A. Further Experimental Results

The further experiments examine student-teacher regression under mild and vast overparameterization. Width 5 can lead to local or global minima, whereas width 45 converges to a global minimum through pruning or teacher-weight matching.

  • A. Further Experimental Results: Width 5 does not reliably find the global minimum, while width 45 converges to a global minimum in the student-teacher regression experiment.The wide network reaches a global minimum by setting some output weights to zero or matching teacher incoming weights up to a ± factor.
  • A. Further Experimental Results: The experiment uses 2D inputs and a two-layer teacher network with r^*=4 sigmoid neurons whose incoming weights are shown as solid black lines.
  • A. Further Experimental Results: The plotted trajectories show incoming-weight paths, convergence locations, and output weights at convergence for the student networks.

A.1. MNIST Experiments for Two-Layer ANNs with Various Widths

The MNIST and combinatorial experiments examine how width affects saddle visits and the relative numbers of critical and global-minimum subspaces. Narrow networks approach saddles more closely, while wider networks show shorter or zigzagging visits, and G(r,m) peaks near r≈0.7m.

  • A.1. MNIST Experiments for Two-Layer ANNs with Various Widths: The MNIST experiments use 10,000 28×28 grayscale images, softplus hidden units, Adam optimization, and full-dataset gradients.
  • A.1. MNIST Experiments for Two-Layer ANNs with Various Widths: Narrow MNIST networks visit saddles for longer, whereas excessive overparameterization produces zigzagging gradient norms consistent with many short saddle visits.
  • A.1. MNIST Experiments for Two-Layer ANNs with Various Widths: G(r,m) is maximized near r≈0.7m for fixed m, providing a width-dependent relationship between critical-subspace counts and network size.
  • A.1. MNIST Experiments for Two-Layer ANNs with Various Widths: For smaller minimal widths, the ratio of critical to global-minimum subspaces decays to zero faster as m increases.The numerical implementation becomes unstable for r^*>35, requiring approximations of G and T at wider minimal widths.
  • A.3. Symmetric Loss Landscape Examples: The symmetric-loss examples show that multiple saddles can occur between two partner global minima on symmetry subspaces.

B.1. Further Properties of Symmetric Losses

Permutation symmetry partitions parameter space into equivalent replicant regions and symmetry subspaces. Gradient flow preserves symmetry subspaces and, for one-dimensional units, remains within a single replicant region without finite-time crossings.

  • B.1. Further Properties of Symmetric Losses: Permutation symmetry creates m! equivalent critical points and partner regions with identical loss, organized by replicant regions defined through unit ordering.
  • B.1. Further Properties of Symmetric Losses: A symmetry subspace formed by k equality constraints has dimension D(m−k+1), with pairwise subspaces H_i,j being the largest.
  • B.1. Further Properties of Symmetric Losses: In one-dimensional unit spaces, paths between replicant regions must cross a symmetry subspace, whereas higher-dimensional units permit paths that avoid them.
  • B.1. Further Properties of Symmetric Losses: Gradient flow initialized on a symmetry subspace remains there, while flow initialized outside all pairwise symmetry subspaces cannot reach one in finite time.
  • B.1. Further Properties of Symmetric Losses: For one-dimensional units, gradient flow stays in its initial replicant region and can therefore be affected only by critical points in that region.

B.2. The Expansion Manifold in Two-Layer ANNs

The expansion manifold is a connected union of affine subspaces generated by expanding an irreducible width-r point to width m. Its subspaces have dimension at least min(d_in,d_out)(m−r), and their number is obtained by counting distinct repeated-weight configurations.

  • Affine-subspace geometry: Θ_r→m(θ_r) consists of distinct non-intersecting affine subspaces whose dimension is at least min(d_in,d_out)(m−r).The dimension bound follows from the free parameters associated with duplicated and zero-type neurons.
  • Counting: The number of affine subspaces is computed by enumerating multiplicity tuples for repeated incoming weights and correcting permutation overcounting with factorial normalization factors.For each configuration, outgoing weights are fixed by the incoming configuration; repeated zero-type groups require additional division by factorials.
  • Connectivity: For m>r, the expansion manifold Θ_r→m(θ_r) is connected, with any two points joined by a union of line segments.The proof proceeds by connecting neuron permutations for m=r+1 and then extending connectivity inductively to larger widths.

B.3. No New Global Minimum

Under a smooth-activation assumption that excludes additional functional invariances, every zero-loss width-m point is obtained by expanding a narrower global minimum. The result does not apply directly to sigmoid, softplus, or tanh, which admit extra cancellations.

  • Assumptions: Assumption A requires σ(0)≠0 and infinitely many nonzero even and odd derivatives at zero, preventing the extra activation-specific cancellations considered later.The proof uses this derivative structure to establish linear independence of the relevant activation combinations.
  • Exceptions: When Assumption A fails, sigmoid, softplus, and tanh permit explicit null-function constructions that enlarge the invariance-generated solution set.For sigmoid, paired opposite arguments already yield cancellations that can be combined into a null function.
  • Main theorem: Under Assumption A, every zero-loss m-neuron point with m>r* belongs to the expansion manifold of a unique narrower global minimum up to permutation.Thus, under the stated activation and data-support conditions, overparameterization creates no additional global minima beyond invariance-generated expansions.

B.4. Symmetry-Induced Critical Points

Expanding an irreducible critical point of width r produces non-intersecting affine critical subspaces at width m. These subspaces preserve strict-saddle structure and are counted by the combinatorial quantity G(r,m).

  • Critical-subspace geometry: The expansion manifold of an irreducible critical point is a union of distinct non-intersecting affine subspaces of dimension m−r, all consisting of critical points.The affine degrees of freedom arise from distributing outgoing weights among repeated neurons while preserving the represented function.
  • Counting: It contains G(r,m) non-intersecting affine subspaces, obtained by summing over positive multiplicity tuples whose entries total m.The count uses distinct configurations of incoming weights because outgoing configurations are determined by them.
  • Saddle structure: If the narrower critical point is a strict saddle, every point in its expansion manifold is also a strict saddle with at least one negative Hessian eigenvalue.The expansion additionally contributes m−r zero eigenvalues because each point lies in an equal-loss affine subspace of dimension m−r.

B.5. Combinatorial Analysis

The combinatorial analysis derives finite-difference representations and closed-form/asymptotic behavior for G and T, clarifying how critical-subspace counts scale across parameterization regimes.

  • G(r,m) equals the r-th forward finite difference of p(x)=x^m for r<m, with G(m,m)=m! and G(r,m)=0 for r>m.
  • The closed forms for T are obtained by reorganizing sums over tuples and applying a counting identity based on occurrence multiplicities.
  • For fixed r, G(r,m) scales asymptotically as r^m when m becomes large.
  • For fixed k and large r, G(r,r+k) has an exact asymptotic relation, and T(m−k,m) grows at the same rate as G(m−k,m).
  • In the mildly overparameterized regime, the analysis derives asymptotics for both G(r*,m) and T(r*,m), while the vastly overparameterized regime is treated through T(r*,m)≥G(r*,m).
Loading 2105.12221v2…