The jamming transition as a paradigm to understand the loss landscape of deep neural networks

Mario Geiger; Stefano Spigler; Stéphane d'Ascoli; Levent Sagun; Marco Baity-Jesi; Giulio Biroli; Matthieu Wyart

Source-linked AI summary

The jamming transition as a paradigm to understand the loss landscape of deep neural networks

Mario Geiger, Stefano Spigler, Stéphane d'Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, Matthieu Wyart

arXiv:1809.09349v4Sep 25, 2018cond-mat.dis-nncs.LG

TL;DR

The paper addresses when fully connected networks avoid poor minima and how their loss landscapes change with depth. It analyzes neural-network training through the jamming transition of repulsive systems and finds a depth-independent fitting threshold, critical Hessian behavior, and avalanche-like learning dynamics near transition.

Problem
The paper asks under which conditions neural networks avoid poor loss minima and how their loss landscapes evolve as depth increases.
Method
The paper analyzes fully connected networks using an analogy between their training-loss landscape and the jamming transition of repulsive ellipses.
Results
Fully connected networks show a parameter-controlled transition between fitting and non-fitting regimes, with fitting ability for random data independent of depth.
Takeaways & Limitations
In the overparametrized regime, too few constraints prevent poor minima, while near transition the landscape has critical curvature and avalanche-like learning dynamics.

Abstract

from arXiv · show

Deep learning has been immensely successful at a variety of tasks, ranging from classification to AI. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in FC networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime, and puts forward the surprising result that the ability of fully connected networks to fit random data is independent of their depth. Our observations suggests that this independence also holds for real data. We also study a quantity $Δ$ which characterizes how well ($Δ<0$) or badly ($Δ>0$) a datum is learned. At the critical point it is power-law distributed, $P_+(Δ)\simΔ^θ$ for $Δ>0$ and $P_-(Δ)\sim(-Δ)^{-γ}$ for $Δ<0$, with $θ\approx0.3$ and $γ\approx0.2$. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

I. INTRODUCTION

The paper asks why non-convex neural-network losses can be optimized successfully and whether depth changes fitting ability. It proposes that fully connected networks undergo a jamming transition controlled by parameter count, with depth-independent fitting of random data.

Neural-network training minimizes a high-dimensional, generally non-convex loss over parameters to fit P training examples.
The paper examines whether deep-learning loss landscapes are glassy and whether training dynamics become trapped in poor minima.
Fully connected networks with cross-entropy or quadratic-hinge loss exhibit a jamming transition separating regimes where all data can or cannot be fitted.
The transition is controlled by the number of parameters N rather than network depth, for both real images and random data.
The transition is sharp and belongs to an ellipse-like universality class, implying many flat directions in the Hessian near criticality.

A. Jamming

Jamming marks a transition from a fluid to a solid as density increases, where the energy landscape and Hessian spectrum become singular. Stability and constraint counting distinguish spherical systems from hypostatic ellipses, while marginal stability produces avalanche-like dynamics and hierarchical minima.

Jamming transition: Jamming occurs when increasing density drives repulsive particles from a fluid into a solid, with finite potential energy appearing in denser packings.For spheres and ellipses, the transition occurs at threshold densities φS and φE.
Landscape curvature: At jamming, the Hessian becomes singular and develops abundant low-energy modes as systems cross from overparametrized to underparametrized regimes.For losses such as cross-entropy, the transition persists but its spectral effects are expected to be less sharp.
Stability and constraints: For spheres, stability and independent constraint counting require the number of contacts to equal the effective degrees of freedom, N∆=N.The argument assumes independent constraints; symmetries can violate this condition.
Hypostatic jamming: Ellipses can jam hypostatically because their second-order Hessian contribution is not negative definite, producing zero-mode structure and a possible spectral gap.The density of vibrational modes includes a delta function at zero with magnitude 1−N∆/N near jamming.
Marginal stability: Marginal stability links power-law distributions of overlaps and gaps to avalanche-like changes in constraints and hierarchical organization of energy minima.Infinite-dimensional calculations report γ=0.41269… and θ=0.42311….

B. Deep Learning

Fully connected networks are framed as a jamming system: increasing effective parameters produces a transition between unsatisfied and fully fitted data, with hypostatic landscape geometry near criticality.

Network setup: A fully connected network maps inputs through weighted neuron layers, with biases included among its parameters and nonlinear activations applied to pre-activations.The architecture has L hidden layers of constant width h; the main activation studied is ReLU, with some tanh results.
Loss functions: The training objective averages a per-datum loss, with cross-entropy and quadratic hinge loss studied for binary classification.For hinge loss, Δ<0 denotes a datum satisfying the classification margin, while N_Δ counts unsatisfied data.
Effective degrees of freedom: The effective parameter count N_eff can be smaller than N when signals fail to propagate, activations are restricted, symmetries occur, or neurons remain inactive.Parameters along N−N_eff directions do not affect the function and generate zero Hessian modes; for properly initialized rectangular fully connected networks, N_eff≈N empirically.
Jamming transition: The jamming transition is defined by increasing N until the trained loss approaches zero, separating an under-parametrized phase with nonzero loss from an over-parametrized phase with null loss.The analysis assumes sufficiently large N permits zero loss while small N leaves positive loss.
Jamming transition: N*≤P/C0, with N*<4P for ReLU, bounds the hinge-loss transition; for N≥P/C0, poor minima cannot be stable because there are too few constraints to form them.The argument uses a nonvanishing fraction C0 of negative eigenvalues of H_p and predicts a hypostatic transition rather than guaranteed isostaticity.
Critical landscape: At the transition, the fraction N_Δ/N of unsatisfied constraints jumps to a finite value, while generically N_Δ/N<1 and the Hessian belongs to the hypostatic universality class.This contrasts with isostaticity, N_Δ/N=1, which cannot be guaranteed for these networks.

III. FOR RANDOM DATA THE TRANSITION OCCURS FOR N ∼P

For random data, fully connected networks reach a fitting transition when the number of parameters scales linearly with the number of patterns. The transition is largely independent of depth, but its measured location depends mildly on training dynamics and finite-time effects.

Transition setup: N* is located by reducing width at fixed depth until the network can no longer fit all random labels within the chosen training time.The transition is defined operationally from a network initially large enough to fit the dataset.
Dynamics dependence: The transition location has mild dependence on optimizer, batch size, learning rate schedule, and other training-dynamics choices.The authors compare this dependence with dynamics-dependent critical densities in particle jamming.
Finite-time effects: Longer training makes the asymptotic relation between N* and P more nearly linear, whereas insufficient training can obscure the transition at large P.For cross-entropy, errors remain at large P and short times, preventing transition identification.
Depth dependence: The transition appears independent of depth and input dimension, indicating that fitting random data depends on parameter count rather than depth in this setup.This behavior was observed across several depths and input dimensions after 10^6 steps.
Loss dependence: Hinge loss produces transition results similar to cross-entropy loss, although the two losses differ in whether training settles at zero loss.Cross-entropy continues drifting at finite training time, which blurs critical behavior near jamming.

IV. THE TRANSITION IS HYPOSTATIC

Near the transition, the fraction of unsatisfied constraints changes discontinuously and is more sharply organized by loss than by the pattern-to-parameter ratio. Misclassification increases smoothly with loss, while the transition also appears for tanh networks.

Constraint transition: NΔ/N jumps from 0 to approximately 0.75 across P/N between 2.8 and 2.9 once the dynamics reaches stationary values.Within this interval, training remains unconverged and the data are scattered, consistent with critical slowing down.
Loss parametrization: Plotting NΔ/N against loss removes nearly all scatter and reveals a discontinuous limit as loss approaches zero.This parallels particle jamming, where potential energy produces less finite-size noise than density.
Misclassification: The number of misclassified data increases monotonically but initially slowly with loss because unsatisfied constraints need not immediately cause misclassification.Patterns with small positive gaps can remain correctly classified despite contributing to the loss.
Activation functions: The sharp transition in NΔ/N also occurs for networks using tanh rather than ReLU activations.This extends the observed transition beyond the ReLU setting.

V. SPECTRUM OF THE HESSIAN OF THE LOSS NEAR JAMMING

Near jamming, the loss Hessian develops the singular and low-energy-mode structure predicted for the transition, with activation-dependent stability features. ReLU cusps can stabilize directions associated with negative Hessian eigenvalues, whereas tanh landscapes are smooth.

Hessian analysis: Near the transition, the Hessian is analyzed on underparametrized points where its curvature is expected to become singular.The study uses hinge loss with ADAM, full batches, orthogonal initialization, and no batch normalization.
ReLU spectra: For ReLU networks, the positive spectrum of Hp collapses after rescaling eigenvalues by L^1/2 and is approximately symmetric across losses and depths.The plotted spectra cover depths L = 2, 3, and 5.
Activation-dependent stability: ReLU cusps may stabilize directions that appear unstable according to the Hessian, so negative eigenvalues do not necessarily imply an unsettled minimum.The alternative explanation is that training has not yet converged locally.
Hessian structure: H0 has a zero-eigenvalue delta peak followed by a gap and continuous positive spectrum, while the total Hessian retains a blurred zero peak and negative eigenvalues.The zero peak in H0 has weight N−NΔ, as expected for hypostatic systems.
Tanh spectra: Tanh networks have a nonsymmetric Hp spectrum with approximately 0.43N negative eigenvalues, and their total Hessian is not gapped at the sampled jamming points.The tanh spectrum differs qualitatively from the approximately symmetric ReLU case.
Across regimes: As N increases toward the overparametrized regime, H0 loses rank but remains sufficient to compensate negative Hp modes, and at L = 0 Hp vanishes.For sufficiently large N, the dynamics reaches a global minimum.

VI. DISTRIBUTION OF GAPS REVEALS NEW SINGULAR BEHAVIOUR

The gap distributions near the transition follow power laws with exponents distinct from particle jamming, and their marginal-stability interpretation connects the loss landscape to hierarchical and avalanche-like dynamics.

Interpretation of gaps: Positive gaps correspond to unsatisfied patterns that increase loss, whereas negative gaps correspond to correctly classified patterns with margin ε.Negative gaps also measure remaining room around learned patterns in the overparametrized regime.
Gap distributions: The positive- and negative-gap distributions follow power laws with θ ≈ 0.3 and γ ≈ 0.2, differing from particle-jamming exponents.For tanh networks, the corresponding exponents are θ ≈ 0.2 and γ ≈ 0.16.
Jamming connection: In particle jamming, saturated stability bounds imply marginal stability, abundant low-energy excitations, and avalanche-like responses; the paper uses this connection to interpret the neural-network transition.The comparison motivates relating critical gap statistics to hierarchical organization and abrupt changes in learned patterns.

VII. IMAGE DATA: MNIST

MNIST exhibits a jamming transition with discontinuous changes in unsatisfied patterns and Hessian spectra resembling those of random data. The transition location is essentially depth-independent but differs from random data, while some critical statistics remain non-universal or poorly resolved.

Data and setup: MNIST uses binary even-versus-odd labels from handwritten digits, with inputs reduced to 10 principal components and trained using hinge loss.The network is fully connected, with L layers of width h and a scalar output.
Jamming transition: The number of unsatisfied patterns shows a discontinuous jamming transition as the number of patterns increases at fixed network size.At the transition, N∆/N is approximately 0.5 rather than the approximately 0.7 observed for random data.
Hessian geometry: The MNIST Hessian spectrum resembles the random-data spectrum, including a symmetric Hp spectrum and a gapped high-energy distribution for H0.The total Hessian is similar to H0, with the zero-mode delta function smeared.
Critical statistics: The negative-gap distribution suggests a power law with exponent γ = 0.25, but the difference from random data is of unclear significance.Positive-overlap statistics vary substantially across samples and were not measured extensively.
Role of depth: The minimum parameter count N* needed to fit MNIST is essentially independent of depth and significantly smaller than for random data.The possible asymptotic scaling of N*(P) for structured data remains an open question.

VIII. CONCLUSION

The paper recasts fully connected network training as a jamming-like constraint problem and argues that its loss landscape separates over- and under-parametrized phases. The transition is associated with singular curvature, depth-independent fitting capacity, and avalanche-like learning dynamics.

Jamming formulation: Replacing cross entropy with hinge loss recasts network training as a continuous constraint-satisfaction problem connected to particle jamming.The paper reports that this loss change does not degrade performance.
Learning dynamics: Near the transition, the distributions P+(∆) ∼ ∆^θ and P−(∆) ∼ (−∆)^−γ have activation-dependent exponents and suggest hierarchical, avalanche-like learning dynamics.The reported ReLU exponents are θ ≈ 0.3 and γ ≈ 0.2; tanh gives θ ≈ 0.2 and γ ≈ 0.16.
Critical behavior: At the transition, the unsatisfied-constraint ratio jumps discontinuously, the Hessian spectrum becomes singular, and the network belongs to the hypostatic universality class.The reported analogy is specifically to repulsive ellipses.
Overparametrized phase: In the overparametrized regime, the analysis supports the absence of poor minima when P/N < P/N* = O(1), because there are too few constraints to prevent minimization.The argument uses a fraction of negative eigenvalues of Hp and treats N as the effective network degrees of freedom.
Under-parametrized phase: In the under-parametrized phase, training remains at positive loss because zero loss is unavailable or the dynamics is trapped in an excited local minimum.The dependence of jamming on dynamics suggests a local minimum in this phase.
Role of depth: For fully connected networks, increasing depth while reducing width at fixed total weights does not reduce the parameters needed to fit random data, and the observation also holds for MNIST.The authors state that broader data studies are needed before generalizing this conclusion.
Implications: The critical line N*(P) is proposed as a reference point for studying landscape geometry, learning dynamics, and generalization.Near threshold, learning may involve abrupt changes in which data are correctly classified.

Appendix A: Effective number of degrees of freedom

The paper defines an effective parameter dimension from local function and gradient variations around training points rather than from raw parameter count. In MNIST experiments, this effective dimension is nearly the total parameter count after accounting for ReLU symmetries.

Motivation: Network symmetries and inactive signal pathways can reduce the effective degrees of freedom below the raw parameter count.If all biases in a ReLU layer are too negative, the network can transmit no signal and have N = 1.
Construction: The effective-dimension construction uses neighborhoods around each training point to avoid a dimension bound determined solely by the number of samples.Function and gradient expansions in pattern space supply the needed local directions.
Construction: For each datum, it suffices to evaluate the point and d coordinate-direction offsets, yielding the effective dimension from the span of the resulting vectors.The construction uses x − xµ = 0, ê1, …, êd, with ê0 = 0.
Empirical result: For three-layer MNIST networks, effective parameters N are nearly equal to total parameters Ñ; the reduction is attributed to one rescaling symmetry per ReLU neuron.The same behavior was observed for random data.

activation function and random data

For ReLU networks with random data, sign symmetry and self-averaging assumptions imply a symmetric spectrum for Hp. The appendix explains this through vanishing odd traces and reports matching pre-activation behavior across training and unseen random data.

Hessian construction: The Hessian contribution Hp is built from pattern-dependent Hessians of the network function and ReLU factors.The appendix analyzes its spectral symmetry in the large-N limit.
Assumptions: The spectral argument assumes that finite powers of Hp self-average over random data and that subleading trace corrections can be neglected.The second assumption is identified as the more difficult one.
Activation statistics: For five-layer random-data networks just above jamming, pre-activation distributions on training and unseen data nearly overlap except for a zero-mass contribution from training data.The zero peak appears only when the training set is included.
Spectral symmetry: Random-label sign symmetry makes odd traces vanish, which implies equal fractions of negative and positive Hp eigenvalues.The conclusion follows after combining the symmetry argument with the self-averaging hypotheses.
Spectral symmetry: Reversing the last-layer weights changes the sign of the relevant odd-order terms while preserving the loss under simultaneous label reversal.The Hessian index structure ensures the required parity under this transformation.

Appendix C: Density of pre-activations for ReLU activation functions

For random data, ReLU pre-activations include a delta distribution at zero, corresponding to loss-function cusp directions; MNIST shows a similar pattern.

For random data, pre-activation densities contain a delta distribution at zero.In a depth-5 network, approximately 0.21N pre-activations are zero.
Approximately 0.21N zero pre-activations occur for random data in a network with L = 5.These zeros correspond to directions in phase space where cusps are present in the loss function.
For MNIST data, the number of zero pre-activations is approximately 0.19N.
For random data with L = 2, the corresponding count is approximately 0.25N.