PaperlayerView exact revision on arXiv

Source-linked AI summary

The Loss Surfaces of Multilayer Networks

Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, Yann LeCun

arXiv:1412.0233v3cs.LG

TL;DR

The paper asks why highly non-convex neural-network losses can nevertheless yield similar results across local minima. It models a fully decoupled multilayer network through its connection to spherical spin-glass Hamiltonians, finding layered low critical values and arguing that global-minimum recovery becomes practically unnecessary.

  • Problem

    The shape and optimization behavior of multilayer neural-network loss functions remain poorly understood, particularly for networks with many parameters.

  • Method

    The paper analyzes a fully decoupled neural network using random matrix theory and a spherical spin-glass model under independence, redundancy, and uniformity assumptions.

  • Results

    For large-size networks, non-diverging-index critical values lie in a band between the global minimum and −ΛE∞(H), with higher points overwhelmingly likely to be high-index saddles.

  • Takeaways & Limitations

    The findings suggest that large networks tend toward low critical points and that recovering the global minimum is practically irrelevant because it may cause overfitting.

  • Takeaways & Limitations

    The theoretical connection relies on a fully decoupled model and assumptions of variable independence, redundancy in parametrization, and uniformity.

Abstract

from arXiv · show

We study the connection between the highly non-convex loss function of a simple model of the fully-connected feed-forward neural network and the Hamiltonian of the spherical spin-glass model under the assumptions of: i) variable independence, ii) redundancy in network parametrization, and iii) uniformity. These assumptions enable us to explain the complexity of the fully decoupled neural network through the prism of the results from random matrix theory. We show that for large-size decoupled networks the lowest critical values of the random loss function form a layered structure and they are located in a well-defined band lower-bounded by the global minimum. The number of local minima outside that band diminishes exponentially with the size of the network. We empirically verify that the mathematical model exhibits similar behavior as the computer simulations, despite the presence of high dependencies in real networks. We conjecture that both simulated annealing and SGD converge to the band of low critical points, and that all critical points found there are local minima of high quality measured by the test error. This emphasizes a major difference between large- and small-size networks where for the latter poor quality local minima have non-zero probability of being recovered. Finally, we prove that recovering the global minimum becomes harder as the network size increases and that it is in practice irrelevant as global minimum often leads to overfitting.

1 Introduction

The paper addresses the poorly understood loss surfaces of multilayer networks by connecting neural-network optimization to random matrix theory and spherical spin-glass models. It argues that large networks tend to have many accessible, similarly performing local minima, while poor minima are more associated with small networks.

  • Motivation: Deep networks use highly non-convex supervised losses whose optimization behavior has historically been unreliable, especially in relatively small networks.Modern systems typically minimize crossentropy or hinge loss with SGD and backpropagation.
  • Motivation: Large networks often have many local minima with similar test performance, suggesting that these minima are relatively easy to find and broadly equivalent.The paper uses random matrix theory to investigate this observed behavior.
  • Approach: Random-matrix results for spherical spin glasses provide a framework for analyzing the critical-point structure of highly non-convex neural-network losses.The paper studies critical points including maxima, minima, and saddle points, whose Hessians may contain many near-zero eigenvalues.
  • Approach: ReLU network losses can be represented as piecewise continuous polynomials whose degree equals network depth and whose monomials correspond to input-output paths.As weights or inputs vary, paths are switched on or off at boundaries between polynomial pieces.
  • Findings: Most local minima in large-size networks are predicted to be equivalent and to yield similar test-set performance.This hypothesis is presented as an empirical claim about learning with large networks.
  • Findings: The probability of finding a bad high-value local minimum is non-zero for small networks but decreases quickly as network size increases.The paper contrasts this with the prevalence of similarly performing minima in large networks.
  • Implications: Recovering the global training minimum is not useful in practice because it may lead to overfitting, whereas training can seek one of many good local minima.The paper frames avoiding saddle points and selecting a suitable attractor as central optimization concerns.
  • Contribution: The paper presents its theoretical description as an early attempt to explain optimization and generalization in neural networks with many parameters, while acknowledging potentially unrealistic assumptions.Its spin-glass connection is described as novel relative to prior work.

3 Deep network and spin-glass model

The paper models a fully connected ReLU network using path-based weights and connects its loss function to the H-spin spherical spin-glass Hamiltonian under independence, redundancy, and uniformity assumptions.

  • Network model: A fully connected ReLU network with one output is represented using layer weights, active paths, network depth H, and layer widths n_i.The input layer is indexed 0, the output layer H, and the activation is σ(x) = max(0, x).
  • Path representation: The network output can be rewritten as a sum over inputs and paths, with each path contributing an input, an activity indicator, and a product of H weights.The path representation introduces γ = n_1n_2...n_H paths and associates each path with an H-weight configuration.
  • Network size: Network mass Ψ counts paths, while network size N counts parameters; with bounded depth, N →∞ is equivalent to Ψ →∞ and Λ →∞.The paper uses these size relationships to connect large networks with the asymptotic spin-glass analysis.
  • Model assumptions: The analysis uses independent inputs and path activities, a spherical weight constraint, and redundancy and uniformity assumptions to simplify the dependent network model.Uniformity requires ordered products of unique weights to appear equally often, while redundancy reduces the number of distinct parameters needed in the representation.
  • Approximation: Under uniformity, the simplified random network output correlates with the original expected output by at least 1/c^2.The correlation guarantee is stated for the variables ˆY_s and Y_s associated with the reduced and original parameterizations.
  • Spin-glass correspondence: After rescaling weights and dropping loss-irrelevant constants, the neural-network loss becomes the H-spin spherical spin-glass Hamiltonian with a spherical constraint.This correspondence allows the paper to use random matrix theory results describing the complexity of spin-glass energy landscapes.

4 Theoretical results

For large-size networks, critical values form a ground-state-bounded band with a layered organization by index. Local minima dominate the low-index region, while high-index saddles lie above the energy barrier and optimization is unlikely to recover distant poor-quality solutions.

  • The existence of the band of low-index critical points: Critical values below −ΛE0(H) are improbable as Λ grows, defining −ΛE0(H) as the ground-state level.Here Λ → ∞ corresponds to increasing network size.
  • The existence of the band of low-index critical points: All non-diverging-index critical values lie in the band (−ΛE0(H), −ΛE∞(H)), while points above its upper threshold are typically high-index saddles.The energy barrier is −ΛE∞(H).
  • Layered structure of low-index critical points: For each fixed k, critical values with index at least k are improbable below −ΛEk(H), where Ek(H) decreases toward E∞.The thresholds satisfy −Ek(H) ∈ [−E0(H), −E∞(H)].
  • Layered structure of low-index critical points: The lowest critical values form successive bands: the first contains only local minima, the next permits index-1 saddles, and higher bands admit progressively larger indices.With overwhelming probability, critical values above the ground state are local minima exclusively in the lowest band.
  • Logarithmic asymptotics of the mean number of critical points: The number of critical points in the main band grows exponentially with Λ, while local minima increasingly dominate saddles and the probability of recovering a saddle there tends to zero.This band is (−ΛE0(H), −ΛE∞(H)).
  • Layered structure of low-index critical points: Low-index minima and saddles occupy the band, whereas high-index saddles lie above the energy barrier; optimizers descend toward low-index points closer to the global minimum.The paper states that finding a bad-quality solution far from the global minimum is highly unlikely for large networks.

5 Experiments

Experiments compare spin-glass and neural-network loss landscapes, finding that increasing network size concentrates solutions in a narrow band of high-quality, low-index minima while weakening the link between training and test loss.

  • Neural Network: 1000 neural networks with 25–500 hidden units were trained on downsampled MNIST using SGD for 200 epochs with learning-rate decay.Each network started from parameters sampled uniformly within the unit cube.
  • Neural Network: 95% parameter redundancy caused less than a 2.5% accuracy drop under simulated annealing, supporting heavy over-parametrization.Weights were restricted to three uniformly spaced values in [−1, 1].
  • Index of critical points: All solutions were minima or very low-index saddle points, with normalized index around 0.01 for the 25-hidden-unit example.The index is the proportion of negative Hessian eigenvalues after eigenvalues below magnitude 0.001 were set to zero.
  • Scaling loss values: As network size increased, scaled neural-network test-loss distributions became less variable and increasingly concentrated near the high-quality critical-point band.Spin-glass experiments showed analogous concentration around −E∞ at larger dimensions.
  • Relationship between train and test loss: Training and test loss became increasingly decorrelated as the number of hidden units grew, limiting the usefulness of seeking the absolute training-loss minimum for generalization.The relationship was measured using Pearson correlation across solutions for each network size.

6 Conclusion

The paper connects fully decoupled large neural-network loss landscapes with spherical spin-glass Hamiltonians under specified assumptions, and finds similar behavior empirically despite dependencies in real networks.

  • 6 Conclusion: The paper establishes a connection between the loss function of a fully decoupled deep network and the Hamiltonian of an H-spin spherical spin-glass model.The connection is derived under assumptions including variable independence, parameter redundancy, and uniformity.
  • 6 Conclusion: Empirical results show similar landscape behavior in the neural-network and spin-glass models despite variable dependencies in real networks.

7 Proof of Theorem 3.1

The proof establishes lower and upper bounds on the quantity N by relating network mass and size and introducing the maximum layer width.

  • 7 Proof of Theorem 3.1: The lower bound on N follows from the arithmetic–geometric mean inequality connecting the mass and size of the network.
  • 7 Proof of Theorem 3.1: The upper bound on N is derived using nmax, the maximum layer width across layers 1 through H.

8 Proof of Theorem 3.2

The proof develops a lemma for two binary classifiers whose accuracies differ by at most ϵ, then specializes it to a network and its reduction image using zero-mean outputs.

  • 8 Proof of Theorem 3.2: Lemma 8.1 considers binary classifiers where the first predicts 1 with probability p ≤ 0.5 and the second differs in accuracy by at most ϵ ∈ [0,p].
  • 8 Proof of Theorem 3.2: The proof partitions the dataset by the first classifier’s predictions and compares the signed outputs of both classifiers.
  • 8 Proof of Theorem 3.2: It computes the expectations, product expectation, and standard deviation of the two signed classifier outputs.
  • 8 Proof of Theorem 3.2: For a network and its reduction image, zero-mean outputs imply p = 0.5, which yields the theorem statement after substitution.

9 Proof of Theorem 3.3

The proof uses centered quantities and the uniformity assumption to establish the stated inequality.

  • The proof notes that both E[ ˆYs] and E[Ys] equal zero.
  • The proof includes an intermediate displayed expression before invoking uniformity.
  • The final inequality follows directly from the uniformity assumption in Equation 6.

10 Loss function as a H - spin spherical spin-glass model

This section formulates neural-network loss terms using random variables and connects them to a spherical spin-glass representation under simplifying assumptions.

  • The model considers random absolute loss and also discusses the hinge loss formulation.
  • The hinge-loss max operator is modeled as an independent Bernoulli random variable M with success probability ρ′.
  • The hinge-loss cases are generalized using S − ˆY when Yt > 0 and S + ˆY when Yt < 0.
  • The derivation further uses Gaussian inputs and a spherical weight constraint while simplifying notation and dropping constants irrelevant to minimization.

11 Asymptotics of the mean number of critical points and local minima

The section extends the asymptotic analysis of critical points and local minima for the spherical spin-glass model as Λ grows.

  • The paper provides asymptotics for the mean number of critical points and local minima as extensions of Theorem 4.1.
  • For H ≥ 3, Theorem 11.1 states an asymptotic result as Λ →∞.
  • Theorem 11.1 uses the Airy function of first kind in its stated expression.
  • Theorem 11.2 gives an asymptotic result for H ≥ 3 and u < −E∞ as Λ →∞.

12 Additional Experiments

Additional experiments examine critical-point indices, optimization methods, scaled test-loss distributions, and how training and test loss vary with network size.

  • 12.1 Distribution of normalized indices of critical points.: For n1 = {10, 25, 50, 100}, all solutions are minima or saddle points of very low index.The normalized index is the proportion of negative Hessian eigenvalues.
  • 12.2 Comparison of SGD and SA.: Figure 6 compares test-loss distributions from SGD and SA across different numbers of hidden units.
  • 12.3 Distributions of the scaled test losses: Figure 8 compares scaled test-loss distributions for spin-glass sizes Λ = {25, 50, 100, 200, 300, 400, 500} and neural-network sizes n1 = {25, 50, 100, 250, 500}.
  • Figure 10 reports the mean and variance of test loss as functions of the number of hidden units.
Loading 1412.0233v3…