Source-linked AI summary
Comparing Dynamics: Deep Neural Networks versus Glassy Systems
M. Baity-Jesi, L. Sagun, M. Geiger, S. Spigler, G. Ben Arous, C. Cammarota, Y. LeCun, M. Wyart, G. Biroli
TL;DR
The paper asks how DNNs navigate complex loss landscapes and whether their training resembles glassy dynamics. It analyzes loss and weight displacements over time and compares DNNs with mean-field glasses. DNNs show aging followed by near-stationary diffusion, while under-parametrized networks display glassy-like behavior, motivating a conjectured distinction between regimes.
Problem
Establishing which account of loss-landscape dynamics applies in realistic DNNs is difficult because critical-point calculations face computational complexity and possible degeneracy.
Source: §1 · p. 2Method
The authors track loss and mean-square weight displacements during training and compare these dynamics with those of the 3-spin spherical glass.
Source: §4 · p. 7Results
DNN training passes through exploration, aging during loss decrease, and near-stationary diffusion, while under-parametrized networks show qualitatively glassy-like displacement behavior.
Source: §4 · pp. 7–8Takeaways & Limitations
The authors conjecture easy and hard phases: over-parametrized networks learn amid many flat directions, whereas under-parametrized networks have rough, glassy dynamics and learn poorly.
Source: §4 · p. 8Barrier crossing occurs on timescales that grow exponentially with system size and is numerically inaccessible except in small systems.
Source: §3 · p. 4
Abstract
from arXiv · showhide
We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.
1. Introduction
The paper examines competing explanations for how neural networks navigate complex loss landscapes, a question that is difficult to settle in realistic models. It tests the analogy with glassy systems by analyzing training dynamics and comparing them with mean-field glasses.
1. Introduction
- DNN training resembles a quench of a disordered system: random weights evolve under SGD toward low loss, analogous to energy descent after a temperature reduction.The dataset plays the role of quenched disorder, with its random coefficients fixed during evolution.Source: §1 · p. 1
- The competing accounts attribute successful training either to descent without barrier crossing through a rough landscape or to the absence of trapping spurious minima.A further account proposes that rare, wide minima have large attraction basins; establishing these scenarios in realistic models is challenging because critical-point calculations are computationally difficult and points may be degenerate.Source: §1 · p. 2
- Prior work also points to saddle points and flat basins as possible features of training paths, while averaging parallel optimizers can reveal wider, possibly rarer basins.The literature offers differing claims about whether gradient-based training avoids saddle-point obstacles.Source: §1 · p. 2
- The authors probe training with one- and two-point correlation functions to infer loss-landscape properties, then compare realistic DNNs with mean-field glasses.Their analysis spans toy architectures and ResNets evaluated on MNIST and CIFAR.Source: §1 · p. 2
2. Basic facts on glassy dynamics
Glassy dynamics is characterized by slow relaxation and aging after a quench. The section introduces energy and two-time displacement as observables, using the p-spin spherical model to illustrate these behaviors.
2. Basic facts on glassy dynamics
- After a quench, the p-spin model’s energy relaxes slowly toward an asymptotic value, unlike the exponential relaxation shown for comparison.Figure 1(a) plots the p = 3 model after a quench from T_i = ∞ to T_f = 0.5.Source: §2 · p. 3, Figure 1 · p. 3
- The two-time correlation measures how much a configuration changes between t_w and t_w + t, and its dependence on both times distinguishes aging from stationarity.For the spherical model, the long-time mean-square displacement approaches two because displacements on the sphere are bounded.Source: §2 · p. 3
- The mean-square displacement depends on both waiting time t_w and elapsed time t; longer waiting times delay escape from its plateau, illustrating aging.The plateau, or Edwards-Anderson parameter, quantifies how frozen the system is in a local minimum.Source: §2 · pp. 3–4
- In glassy systems, slow dynamics after a quench is attributed not to barrier crossing but to increasingly flat directions that slow gradient descent.At long times, a decreasing number of negative Hessian eigenvalues accompanies the slowdown.Source: §2 · p. 4
3. Models and Results
Across varied network architectures and datasets, training exhibits distinct loss and mean-square-displacement regimes. The dynamics slow as flat directions emerge, then approach diffusion near the landscape bottom, with late-time behavior differing from the p-spin model.
3. Models and Results
- The evaluation spans toy, fully connected, convolutional, and ResNet architectures trained on MNIST, CIFAR-10, and CIFAR100 across multiple parameter sets.The models range from about 6 × 10^4 to 3 × 10^8 weights.Source: §3 · p. 4
3.1. The Loss Function
- Training has three loss regimes: an initial plateau, a roughly linear loss decrease with rising accuracy, and sharply slower decay as loss approaches its minimum.The transition times t1 and t2 are also identifiable through mean-square displacement; the crossover is less sharp in the toy-model plot.Source: §3.1 · p. 4, Figure 2 · p. 5
- The slowing is attributed not to barrier crossing but likely to increasingly many flat directions, and the loss ultimately approaches its lowest possible value.The p-spin model instead remains in higher, wider minima on the described timescales.Source: §3.1 · p. 5
3.2. Further evidence: Two-time correlation functions
- Mean-square displacement tracks the loss regimes: it collapses for tw < t1, ages for t1 < tw < t2, then has a characteristic time that no longer grows with tw.During the aging regime, the characteristic time increases with waiting time before the displacement becomes flat for t > t2 − tw.Source: §3.2 · p. 6, Figure 3 · p. 6
- For tw > t2, curves collapse when scaled by D(tw), indicating stationary dynamics apart from changing noise as the near-zero-loss system diffuses near the landscape bottom.The plotted panels separate tw < t2 from tw > t2; D(tw) normalization accounts approximately for changing gradient noise.Source: §3.2 · p. 6, Figure 4 · p. 7
- Unlike DNN training, p-spin aging persists as energy approaches its asymptotic value, and DNN mean-square displacements lack the p-spin short-time collapse.The comparison concerns time scales that do not diverge with system size.Source: §3.2 · p. 6
- Aging and diffusive regimes appear qualitatively across the analyzed networks, including the toy model described as having no barriers.This supports the authors’ interpretation that slowing and late diffusion are associated with flat directions.Source: §3.2 · p. 7
4. Discussion
Training passes through exploration, loss reduction with aging, and a final nearly stationary, diffusive regime; its slowdown is attributed to increasingly many flat directions, not barrier crossing. The discussion contrasts DNN and glassy-system landscape statistics and reports glass-like dynamics when model A is reduced to 10 hidden neurons.
4. Discussion
- Training has three regimes—initial landscape exploration, loss decrease with aging, and a nearly stationary, diffusive final phase—with slowdown traced to increasingly many flat directions, not barrier crossing.The final diffusion occurs as the dynamics approaches the bottom of the landscape.Source: §4 · pp. 7–8
- With 10 hidden neurons on CIFAR-10, loss does not reach zero and mean-square displacement shows glass-like collapse followed by waiting-time-dependent aging.The figure plots train/test loss and accuracy against log(t), with mean-square displacement in a separate panel.Source: Figure 5 · p. 8, §4 · p. 8
- DNNs and mean-field glassy systems both slow along almost-flat directions, but DNN dynamics reaches the landscape bottom and suggests different landscape statistics.The authors propose over-parameterization as a possible reason for the difference.Source: §4 · p. 8
- The authors conjecture an easy over-parameterized phase and a hard under-parameterized phase, while noting that detailed investigation remains future work.The proposed transition is compared with phase behavior in combinatorial optimization and the emergence of flat directions at the jamming transition.Source: §4 · p. 8