Essentially No Barriers in Neural Network Energy Landscape

Felix Draxler; Kambis Veschgini; Manfred Salmhofer; Fred A. Hamprecht

Source-linked AI summary

Essentially No Barriers in Neural Network Energy Landscape

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, Fred A. Hamprecht

arXiv:1803.00885v5Mar 2, 2018stat.MLcs.AIcs.LG

TL;DR

Because theoretical insight into successful training remains limited for high-dimensional, non-convex neural-network losses, this paper characterizes connections between minima using minimum-energy paths. It finds that minima form a connected component through constantly low-loss paths, with especially low barriers as depth and width increase, while test loss remains essentially constant and test error slightly increases.

Problem
Theoretical insights into why neural networks train successfully despite high-dimensional, non-convex loss functions remain limited or rely on strong assumptions.
Method
The paper uses neural-network minimum-energy-path methods, including NEB, to bend paths between minima until perpendicular gradient forces vanish and to estimate saddle-point energy upper bounds.
Results
Minima form one connected component through constantly low-loss paths, with especially low barriers as network depth and width increase; test loss remains essentially constant while test error slightly increases.
Takeaways & Limitations
The findings imply that low Hessian eigenvalues exist beyond analytically zero eigenvalues caused by scaling and motivate further theoretical work on neural-network loss surfaces.
Takeaways & Limitations
The authors cannot formally characterize the regime where barriers disappear, and a proof is complicated by dependence on parameters, architecture, training data, and structured-data distributions.

Abstract

from arXiv · show

Training neural networks involves finding minima of a high-dimensional non-convex loss function. Knowledge of the structure of this energy landscape is sparse. Relaxing from linear interpolations, we construct continuous paths between minima of recent neural network architectures on CIFAR10 and CIFAR100. Surprisingly, the paths are essentially flat in both the training and test landscapes. This implies that neural networks have enough capacity for structural changes, or that these changes are small between minima. Also, each minimum has at least one vanishing Hessian eigenvalue in addition to those resulting from trivial invariance.

1. Introduction

The paper challenges the view of isolated minima in finite-width valleys, proposing that low-loss regions of modern neural-network landscapes are connected. Experiments find continuous, essentially flat paths between independently trained minima, with nearly unchanged training and test loss.

Motivation and hypothesis: The authors conjecture that parameter regions below a low loss threshold form one connected component rather than isolated minima.This contrasts with the conventional picture of minima as points in strictly convex valleys.
Approach: They study random pairs of minima in ResNets and DenseNets trained on CIFAR10 and CIFAR100, constructing continuous low-loss paths through parameter space.The paths connect minima reached from different initialisations, beyond known loss-conserving rescaling operations.
Findings: The test loss remains essentially constant along these paths, while the test error rate increases slightly.This indicates that the flat-path phenomenon appears in both training and test landscapes, with a small classification-accuracy trade-off.
Interpretation: The observed paths suggest that modern networks have enough parameters for substantial structural changes while retaining good predictions.The authors present this qualitative interpretation as a basis for future theoretical investigation.

2. Related Work

The paper situates its contribution against earlier work on connected minima and degeneracy. It applies AutoNEB to higher-performing ResNets and DenseNets on CIFAR10 and CIFAR100, finding essentially flat connections between their minima.

Prior landscape views: Earlier work pictured minima as finite-width valleys whose width was associated with generalisation, whereas this paper emphasizes low-loss paths through parameter space.The alternative landscape picture had also been conjectured in prior work.
Degeneracy: Prior analyses identified degeneracy from layerwise rescaling, while this work studies a different degeneracy connecting independent minima through essentially flat paths.The distinction separates known output-preserving symmetries from the paper’s empirical path construction.
Connected minima: Freeman and Bruna found low-barrier connections for a CNN on MNIST and an RNN on PTB but a more frustrated landscape on CIFAR10.This paper extends that line of inquiry to modern residual and densely connected architectures.
NEB studies: Earlier NEB studies found high barriers in small multilayer perceptrons that disappeared with wider hidden layers, while regularisation also removed significant barriers in another MNIST study.The reported error rate in the latter study was 14.8%, compared with 12% for a linear classifier and 0.35% for a standard CNN.
Current contribution: This work applies AutoNEB to a nontrivial network and reports essentially flat paths between minima of state-of-the-art networks on CIFAR10 and CIFAR100.An independent contemporaneous study reported the same broad observation across several architectures and both datasets.

3. Method

The method approximates minimum-energy paths between converged neural-network minima using NEB and AutoNEB, while addressing path sampling and computational limitations. It uses adaptive pivot redistribution and graph-based exploration to estimate connections among multiple minima.

Minimum Energy Path: The objective is to find a continuous path between two converged minima whose maximum loss is as low as possible.Such a path is called a minimum energy path (MEP), and its highest-loss parameter set is treated as the path’s saddle point.
Nudged Elastic Band: NEB bends an initial straight path using gradient forces until gradients perpendicular to the path vanish, yielding a critical path whose highest point upper-bounds the saddle-point energy.The loss force acts perpendicularly to the path, while the spring force acts parallelly to redistribute pivots without straightening the path.
Nudged Elastic Band: The NEB path uses N internal pivots between fixed endpoints θ1 and θ2, with spring stiffness k and gradient-based updates controlling the optimization.The algorithm redistributes pivots, updates them using the NEB force, and repeats for T iterations until convergence.
Limitations: The choice of spring stiffness k creates a trade-off: values that are too small undersample high-energy regions, while values that are too large straighten paths and can miss saddle points.The authors also report that no single k worked well across different loss surfaces.
AutoNEB: AutoNEB approximates MEPs in neural networks by iteratively applying NEB and adding pivots where the current path is insufficiently sampled.The approach is used because exact MEP construction is infeasible in parameter spaces with millions of dimensions.
Local Minimum Energy Paths: For multiple minima, AutoNEB explores local MEPs through a minimum spanning tree and uses path concatenation to obtain upper bounds on unknown pairwise saddle-point energies.The procedure can stop early when the spanning tree contains only similar saddle-point energies, reducing the cost of exhaustive exploration.

4. Experiments

The experiments use AutoNEB to connect distinct minima across CNN, ResNet, and DenseNet architectures on CIFAR10 and CIFAR100. Saddle losses become nearly indistinguishable from minima losses in deep networks while test accuracy is largely preserved.

Experimental setup: AutoNEB connects minima from different random initialisations and evaluates saddle losses using a minimum-spanning-tree reduction.Ten minima are considered per architecture, with paths optimized over 14 NEB cycles.
Saddle point losses: The saddle losses are small for shallow networks and almost negligible for deep residual networks.Across architectures, saddle losses are compared with both training and test losses.
Saddle point losses: For deep architectures, saddle loss lies close to training loss, while minimum and saddle losses are very close on the test set.On CIFAR100, a small training-loss gap remains but is negligible relative to a randomly initialized network.
Learning dynamics: The training loss crosses the saddle energy only late in optimization, after a learning-rate decay and an additional significant loss drop.For many wider CNNs, ResNets, and DenseNets, the crossing occurs after the second decay.
Saddle point losses: Deeper and wider architectures have lower barriers, which essentially vanish in current-day deep networks; CIFAR100 leaves higher barriers than CIFAR10.Increasing depth, especially, closes the gap between saddle and minimum losses in simple CNNs.
Test performance: The classification error increases by at most 0.5% on CIFAR10 and 2.2% on CIFAR100 for deep architectures relative to the minima.Thus, the low-loss paths preserve test accuracy within the reported bounds.
Path structure: Network parameters vary smoothly along local MEPs, with the largest deviations from linear interpolation near the saddle point.The paths are between 50% and 2.5 times longer than direct connections between minima.

5. Discussion

The discussion interprets low barriers as evidence of resilience and redundancy in neural networks. Extra capacity can permit continuous, loss-preserving structural changes, although the empirical regime lacks a formal characterization.

Barrier interpretation: The empirical saddle-loss upper bounds are astonishingly close to the losses at the minima, especially as networks become wider and deeper.The disappearance of barriers emerges most strongly with increased width and depth.
Limitations: The paper cannot formally characterize the regime where barriers disappear or provide a proof because the loss depends on architecture, training data, and high-dimensional structured datasets.This is an explicit scope limitation of the discussion.
Redundancy: In the XOR example, exchanging two hidden neurons creates minima whose continuous transition necessarily misclassifies at least one instance.The obstruction arises despite the two networks being equivalent minima.
Redundancy: Adding one auxiliary neuron enables a zero-loss transition by temporarily replacing one neuron and disabling its outgoing contribution during the switch.The construction demonstrates how spare width supports a loss-preserving path.
Redundancy: Adding a second two-neuron layer likewise permits a zero-loss path, showing that extra depth can provide the needed degrees of freedom.Both width and depth are presented as mechanisms for constructing continuous low-loss transitions.

6. Conclusion

The conclusion presents deep-network minima as connected by constantly low-loss paths rather than isolated finite-width valleys. It introduces AutoNEB as a tool for studying this landscape and notes possible ensemble and theoretical uses.

Conclusion: Deep neural-network minima form one connected component through paths with constantly low loss, with barriers especially low as depth and width increase.This summarizes the paper’s central landscape conclusion.
Conclusion: The proposed explanations require spare network capacity or degrees of freedom, which appears empirically available in modern architectures on standard problems.The conclusion limits this interpretation to the stated empirical setting.
Hessian structure: Low Hessian eigenvalues exist beyond the analytically zero eigenvalues caused by scaling invariance.This is presented as an implication of the observed landscape structure.
Future work: AutoNEB opens further empirical study of neural-network energy landscapes and may support clustering minima through saddle barriers and disconnectivity graphs.The authors expect refined hyperparameters could recover lower, potentially true saddle paths.
Practical implications: The resulting paths may serve as large ensembles of neural networks because the authors observe marginally lower test loss along them.This is an envisaged practical use rather than a demonstrated deployment result.

A. AutoNEB insertion

AutoNEB refines piecewise-linear paths by inserting pivots where the true loss deviates substantially from the interpolated estimate. It prioritizes the largest deviations while limiting insertions per iteration.

A. AutoNEB insertion: New pivots are inserted where evaluated loss exceeds the estimate from adjacent pivot values by a threshold.The loss between pivots is approximated by interpolating their values, with α ∈ [0, 1] denoting position along the segment.
A. AutoNEB insertion: The true-minus-estimated loss difference is evaluated at discrete positions α ∈ A ⊂ (0, 1) and normalized to the range of pivot losses.
A. AutoNEB insertion: A new pivot is inserted between pivots i and i + 1 when the normalized deviation exceeds threshold ϑ.
A. AutoNEB insertion: Only one pivot is inserted per segment per iteration, selecting the highest-residuum position and prioritizing the largest deviations globally.

B. Quantitative minimum and saddle point losses

Across the reported architectures, saddle-point losses remain substantially below initial loss and close to minima, while classification error changes little along the paths.

B. Quantitative minimum and saddle point losses: The untrained-network loss is 2.3 on CIFAR10 and 4.6 on CIFAR100.
B. Quantitative minimum and saddle point losses: Saddle point energies range from one-third to one-twentieth of initial loss for shallow networks and are about two orders smaller for deep residual networks.
B. Quantitative minimum and saddle point losses: Test saddle losses approach minimum test losses as CNNs become wider or deeper, with especially small values for deep residual networks and ResNets.On CIFAR10, deep residual-network saddle energies are about one order of magnitude below average minimum test loss; on CIFAR100, ResNet values are below one-third of test loss.
B. Quantitative minimum and saddle point losses: Training saddle loss is at most eight times the training loss at minima and is closer to training than test loss for all but three smallest basic CNNs.For deeper architectures, saddle loss is much closer to training than test loss.
B. Quantitative minimum and saddle point losses: Classification error rises by at most 0.7% on CIFAR10 and 2.9% on CIFAR100 for ResNets, versus 0.4% and 1.5% for DenseNets.These increases are small compared with error rates at the minima.