Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

Lei Wu; Zhanxing Zhu; Weinan E

Source-linked AI summary

Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes

Lei Wu, Zhanxing Zhu, Weinan E

arXiv:1706.10239v2Jun 30, 2017cs.LGcs.AIstat.ML

TL;DR

Deep learning often generalizes despite over-parameterization, but the reasons and the distinction between equally fitting good and bad minima remain unclear. The paper analyzes loss landscapes theoretically for 2-layer networks and numerically for deeper networks, finding that good minima have larger attraction basins associated with low-complexity, small-Hessian solutions. It concludes that random initialization therefore tends to lead optimization toward good minima, while deeper-network theory remains incomplete.

Problem
The paper addresses why over-parameterized deep networks generalize well and what distinguishes minima with the same small training error but different generalization performance.
Method
The paper analyzes 2-layer neural-network loss landscapes theoretically and studies deeper networks through numerical experiments, using Hessian-based characterization of attraction-basin volume.
Results
Good minima have larger attraction basins, while low-complexity solutions have small Hessian norms; numerical evidence supports their relationship with better generalization.
Takeaways & Limitations
Loss-landscape structure is proposed to make random initialization converge to good minima with high probability, independently of the optimizer type.
Takeaways & Limitations
The theoretical analysis provides only a rough, hypothesis-dependent account for 2-layer networks, and deeper networks lack a full theoretical justification.

Abstract

from arXiv · show

It is widely observed that deep learning models with learned parameters generalize well, even with much more model parameters than the number of training samples. We systematically investigate the underlying reasons why deep neural networks often generalize well, and reveal the difference between the minima (with the same training error) that generalize well and those they don't. We show that it is the characteristics the landscape of the loss function that explains the good generalization capability. For the landscape of loss function for deep networks, the volume of basin of attraction of good minima dominates over that of poor minima, which guarantees optimization methods with random initialization to converge to good minima. We theoretically justify our findings through analyzing 2-layer neural networks; and show that the low-complexity solutions have a small norm of Hessian matrix with respect to model parameters. For deeper networks, extensive numerical evidence helps to support our arguments.

1 Introduction

The paper asks why over-parameterized deep networks generalize well and what distinguishes equally fitting good and bad minima, arguing that loss-landscape geometry explains both phenomena.

Motivation: Deep networks can generalize well despite having more effective parameters than training samples, challenging conventional expectations for over-parameterized non-convex models.The paper frames this as part of deep learning’s theoretical black-box problem.
Problem: “Good” and “bad” minima have the same small training error but differ substantially in generalization performance, measured by test accuracy.The distinction concerns generalization, not training fit.
Problem: On MNIST, minima with the same 100% training accuracy can exceed 95% test accuracy or perform like a random guess.The paper accesses rarely observed bad solutions by intentionally adding attacking data to the training set.
Research questions: The study investigates which property distinguishes well-generalizing minima and why random-initialized optimizers almost surely reach them.These are the paper’s two central questions.
Contributions: Large-attractor-volume minima often generalize better, while loss-landscape characteristics—not optimizer type—are conjectured to guide random initialization toward good minima.The paper contrasts this explanation with accounts emphasizing SGD or regularization.
Related work: Prior work largely compared small differences among good solutions, leaving the substantial good-versus-bad-minima distinction insufficiently addressed.The paper positions its analysis as addressing this gap.

2 Deep neural networks learn low-complexity solutions

Deep networks can fit training data with low-complexity solutions despite high capacity, while other zero-error solutions generalize poorly. Compared with kernel regression, optimization in FNNs tends to find low-complexity solutions without explicit regularization.

2 Deep neural networks learn low-complexity solutions: Empirical risk is minimized over a hypothesis space, with each hypothesis often parameterized by neural-network parameters θ.The empirical risk averages the loss over N i.i.d. training samples.
2 Deep neural networks learn low-complexity solutions: With the same small training error, simpler input-output mappings generalize better than more complex solutions.This follows the paper’s complexity-based interpretation of generalization error.
2.1 Optimizers converge to low-complexity solutions: A five-point fitting experiment found that FNN optimizers converged to low-complexity solutions across different depths, including a 12-layer network with about 18000 parameters.The high-capacity network still generalized well despite having far more parameters than training points.
2.1 Optimizers converge to low-complexity solutions: High-complexity FNN solutions could also be constructed by intentionally attacking the training set, showing that optimization did not merely reflect a simple hypothesis space.The contrasting overfitting solution is shown as a dashed line in Figure 1.
2.1 Optimizers converge to low-complexity solutions: Unlike FNN optimization, kernel ridge regression produced overfitting solutions as polynomial-kernel capacity increased, requiring regularization to control solution complexity.The comparison used different polynomial-kernel orders and a regularization parameter of 0.01.
2.2 Connection with classical learning theory: The high-capacity solution set contains diverse generalization performance, yet random-initialized optimizers rarely converge to its bad solutions.Some solutions generalize no better than a random guess, so whole-space complexity bounds do not explain this optimizer behavior.

3 Perspective of loss landscape for understanding generalization

The paper argues that loss-landscape geometry, rather than SGD noise alone, guides optimizers toward low-complexity solutions because good minima occupy overwhelmingly larger attraction basins.

Landscape perspective: The proposed mechanism is that loss-landscape geometry guides optimizers toward low-complexity solutions.The paper investigates stochasticity, implicit regularization, and landscape characteristics, but focuses on the landscape as the central explanation.
Basins of attraction: Good minima have attraction basins whose volume dominates that of poor minima, making random initialization overwhelmingly likely to enter good basins.The paper formalizes basins of attraction for minima and relates initialization probabilities to basin volumes.
SGD is not the magic: SGD improves test performance over full-batch gradient descent, but the improvement is limited and cannot alone explain deep networks’ generalization.The comparison uses three MNIST networks trained with the two optimization methods.
Random initialization: Across initialization strategies, the experiments report convergence to good solutions, supporting the claim that random initialization favors good basins.The tested strategies include uniform and Gaussian initializations, with six runs for each network and strategy.

4 Landscape of 2-layer networks

The 2-layer analysis connects hypothesis complexity to parameter-space curvature, arguing that low-complexity solutions lie in flat, large basins while high-complexity solutions lie in sharp, small ones.

Landscape characterization: The 2-layer analysis studies whether low-complexity solutions occupy flat regions with large attraction basins.The analysis uses the loss landscape and basin geometry to distinguish solutions with different generalization behavior.
Hessian decomposition: The empirical-risk Hessian is decomposed into Fisher-information and fitting-residual terms under least-square loss.The population Fisher information is defined using parameter gradients of the network output.
Complexity and Fisher information: Theorem 1 relates hypothesis complexity, measured through expected input-gradient norm, to the Fisher information matrix for model parameters.The Fisher information is then connected to the Hessian of empirical risk.
Curvature bound: For sufficiently many samples, the Monte Carlo approximation term can be ignored, supporting a curvature-based characterization of the landscape.For ReLU networks, the activation-curvature term vanishes, so small-norm hypothesis complexity is bounded by the Hessian’s Frobenius norm.
Complexity and basin size: Under a node-scaling-preserving constraint, low-complexity solutions correspond to small-Hessian, flat, large-basin regions, whereas high-complexity solutions correspond to sharp, small basins.The paper argues that near-zero random initialization implicitly explores a region where bad solutions have much smaller attraction basins.
Scope: The theoretical analysis is limited to a rough hypothesis-dependent account for 2-layer networks, while more elaborate analysis for deeper networks remains future work.The paper supports deeper-network arguments with numerical experiments rather than the same theoretical treatment.

5 Experiments

The experiments construct solutions with diverse generalization performance and examine whether Hessian-based basin measures track test accuracy. Across small and large networks, good minima occupy larger valleys, although the Hessian-based relationship has variance.

5.1 Construct solutions with diverse generalization performance: An attack dataset with intentionally wrong labels generates diverse solutions, including minima with low training error but severely harmed test performance.Tuning the attack weight γ and attack-set size produces solutions with very bad generalization.
5.2 Spectral analysis of Hessian matrices: The attractor basin is difficult to estimate directly, so local flatness and Hessian spectra are used as proxies for basin volume.A large basin is associated with a flat local valley, while a small basin is associated with a sharp one.
5.3 Numerical evidence for deep neural networks: Small mLeNets are trained on MNIST and SVHN, while ResNet-32 is trained on CIFAR-10 using attack sets to produce varied solutions.For large networks, the Frobenius norm of the Hessian is estimated statistically because computing its spectrum is prohibitively expensive.
5.3 Numerical evidence for deep neural networks: Good minima are located in very large valleys, whereas bad minima occupy small valleys; the bad-to-good basin-volume ratio can be exponentially small.This supports the claim that random initialization makes optimization converge to good solutions almost surely.
5.3 Numerical evidence for deep neural networks: The relationship between generalization error and basin volume shows variance, and V(k) may fail to distinguish minima with equivalent generalization performance.The paper attributes this limitation to the roughness of the Hessian-based volume estimate and suggests a better non-local quantity is needed.

6 Conclusion

The paper connects generalization to loss-landscape geometry: low-complexity solutions have smaller Hessian norms and larger attractor basins. Numerical experiments extend the analysis to deeper networks, while theoretical justification there remains challenging.

6 Conclusion: Two-layer analysis shows that low-complexity solutions have a small Hessian norm with respect to model parameters.This distinguishes minima that generalize well from those that generalize poorly.
6 Conclusion: For deeper networks, numerical experiments support the relationship between Hessian structure, basin volume, and generalization.The paper states that theoretical justification for deeper networks remains a challenge.
6 Conclusion: Good minima have basin volumes dominating those of poor minima, leading to almost-sure convergence to good solutions under random initialization.The conclusion presents this as the consequence of the Hessian-implied landscape structure.

Supplementary Materials

The supplementary materials describe the network architectures used for Hessian calculations, optimization experiments, and standard image-classification evaluations.

Model details: mLeNet is a modified LeNet designed to make repeated Hessian computation practical, with 3,781 parameters.Its final fully connected layer is replaced by a convolutional layer and global average pooling.
Model details: sNIN is a small Network in Network model used for full-batch gradient-descent experiments.Its three block feature counts are set to (20, 30, 20) because of GPU-memory limits.
Model details: ResNet-18 and ResNet-32 are standard models using batch normalization.

Appendix B: landscape of convex model

The convex-model appendix explains why empirical-risk information alone may not distinguish good and bad solutions, unlike the loss landscapes of deep neural networks.

Appendix B: landscape of convex model: In overparameterized shallow models, good and bad solutions can share the same empirical-risk information and remain indistinguishable from that risk alone.The example uses θ ∈ R^d with d > N and a feature map φ(x).
Appendix B: landscape of convex model: At a global minimum, the empirical risk can have identical first- and second-order information across different solutions.For losses such as L2 and hinge loss, the relevant loss derivative is constant at correct prediction.
Appendix B: landscape of convex model: Because optimizers cannot distinguish these convex-model minima through empirical risk, proper regularization is needed to steer them toward low-complexity solutions.
Appendix B: landscape of convex model: Deep neural networks differ because non-convexity separates good and bad solutions into distinct valleys distinguishable through empirical risk.

Proof of theorem 1

The passage presents a symbolic expression from the proof of theorem 1, but does not state its interpretation or conclusion.

The displayed fragment includes indexed quantities and functions of x.
The passage does not provide enough prose to identify the expression’s role in the proof.

Proof of corollary 1

The corollary proof invokes the central limit theorem, assumes a bounded second derivative for the activation function, and combines preceding equations to characterize Remp.

The proof invokes the central limit theorem before presenting a derived relation.
It assumes the activation function’s second derivative is bounded, with finite Cσ := ∥σ′′(x)∥∞.The passage states that sigmoid and tanh are among the commonly used activations satisfying this assumption.
Defining D = maxk |∥bk∥2ak|, the proof combines Eqs. 18 and 19 to characterize Remp.