Source-linked AI summary
Sharp Minima Can Generalize For Deep Nets
Laurent Dinh, Razvan Pascanu, Samy Bengio, Yoshua Bengio
TL;DR
The paper asks whether flatness of loss minima explains why deep networks generalize, and argues that common flatness notions are problematic for deep models. It exploits parameter-space symmetries and reparametrizations in rectified networks to show that equivalent functions can correspond to different flatness properties without changing generalization. The paper concludes that flatness must be defined more carefully to avoid degeneracies.
Problem
The paper addresses the open question of why highly overparameterized deep networks generalize well, focusing on the hypothesis that flat minima lead to better generalization.
Method
The paper analyzes non-identifiability, non-negative homogeneity, α-scale transformations, and parameter reparametrization in deep rectified networks.
Results
Equivalent rectified-network models can have arbitrarily altered flatness measures, while representing the same function and retaining the same generalization behavior.
Takeaways & Limitations
Flatness measures should account for architectural symmetries and parametrization because parameter geometry alone need not reflect a function's generalization properties.
Takeaways & Limitations
The analysis primarily uses deep rectified networks, excludes bias terms for convenience, and does not establish the same issue for random-subspace ϵ-sharpness.
Abstract
from arXiv · showhide
Despite their overwhelming capacity to overfit, deep learning architectures tend to generalize relatively well to unseen data, allowing them to be deployed in practice. However, explaining why this is the case is still an open area of research. One standing hypothesis that is gaining popularity, e.g. Hochreiter & Schmidhuber (1997); Keskar et al. (2017), is that the flatness of minima of the loss function found by stochastic gradient based methods results in good generalization. This paper argues that most notions of flatness are problematic for deep models and can not be directly applied to explain generalization. Specifically, when focusing on deep networks with rectifier units, we can exploit the particular geometry of parameter space induced by the inherent symmetries that these architectures exhibit to build equivalent models corresponding to arbitrarily sharper minima. Furthermore, if we allow to reparametrize a function, the geometry of its parameters can change drastically without affecting its generalization properties.
1 Introduction
The paper examines why stochastic-gradient solutions generalize well, focusing on the hypothesis that flat minima improve generalization. It argues that common flatness notions are not directly reliable for several deep architectures because parameter-space geometry and symmetries can change flatness without changing the represented function.
- The paper focuses on estimation error and the question of why stochastic gradient descent produces solutions that generalize well.
- The flat-minima hypothesis proposes that minima with broad, low-error neighborhoods generalize better than sharp minima.Different works define flatness through connected-region size, curvature, or maximum loss in a bounded neighborhood.
- Common flatness measures can alter the ranking of prediction functions because the geometry of parameter space depends on architecture and parametrization.
- Parameter changes can modify flatness without changing the represented function, whereas KL divergence is invariant to reparametrization.
- The paper concludes that flatness requires more careful definitions to avoid degeneracies in deep models.
2 Definitions of flatness/sharpness
The paper formalizes several relative notions of flatness and sharpness around a loss minimum. These include connected low-loss volume, Hessian-based curvature, and worst-case loss within an Euclidean neighborhood.
- Figure 1 contrasts volume ϵ-flatness, represented by horizontal width at loss height ϵ, with ϵ-sharpness, represented by height over width 2ϵ.
- These measures compare minima relatively, because flatness or sharpness is defined in relation to a chosen neighborhood, curvature summary, or loss threshold.
- Volume ϵ-flatness is the volume of the largest connected region containing a minimum where loss remains below L(θ) + ϵ.
- Hessian-based flatness summarizes local curvature using scalar measures such as the spectral norm or trace of the Hessian.
- Keskar et al.'s ϵ-sharpness measures the loss increase within an Euclidean ball B2(ϵ, θ) around a minimum.
3 Properties of Deep Rectified Networks
Deep rectified networks possess non-negative homogeneity and observationally equivalent parameter configurations. This non-identifiability creates parameter-space directions that preserve predictions while changing the geometry used to measure flatness.
- 3 Properties of Deep Rectified Networks: The paper studies deep rectified feedforward networks with a linear output layer, while noting that the arguments extend to other architectures.
- 3 Properties of Deep Rectified Networks: Rectified activations apply the elementwise positive-part function, mapping each z_i to max(z_i, 0).
- 3 Properties of Deep Rectified Networks: The setup excludes bias terms for convenience, although the authors state that the arguments extend to models including biases.
- 3 Properties of Deep Rectified Networks: Because parameter changes can preserve model behavior, the parameter-space metric is nonconstant and can differ from the geometry of changes in the represented function.
- 3 Properties of Deep Rectified Networks: Non-negative homogeneity creates infinitely many observationally equivalent parameter settings that produce the same prediction function.
- 3 Properties of Deep Rectified Networks: The paper uses α-scale transformations that preserve the represented function and therefore do not affect generalization.
4 Deep Rectified networks and flat minima
Rectified networks have parameter symmetries that preserve predictions while changing geometric flatness measures, undermining flatness as a direct explanation of generalization. In particular, volume flatness can become infinite, Hessian-based sharpness arbitrarily large, and epsilon-sharpness high for equally generalizing minima.
- Parameter symmetries: α-scale transformations preserve the represented function while significantly decreasing several flatness measures; for one measure, all minima become equally flat.The transformation rescales adjacent parameter components as (θ1, θ2) → (αθ1, α^-1θ2).
- 4.1 Volume ϵ-flatness: For any nonconstant one-hidden-layer rectified network minimum, C(L, θ, ϵ) has infinite volume for every ϵ > 0.A nonzero-volume neighborhood is transformed into disjoint regions whose volumes can grow without bound when layer dimensions differ.
- 4.1 Volume ϵ-flatness: Consequently, every rectified-network minimum is infinitely flat under volume ϵ-flatness, so this measure cannot gauge a minimum’s generalization property.The argument generalizes to rectified neural networks in general.
- 4.2 Hessian-based measures: The same transformations can increase several Hessian eigenvalues, including in deeper rectified networks, without changing the function’s behavior.The paper examines both the Hessian spectral radius and trace as manipulable flatness measures.
- 4.2 Hessian-based measures: The gradient and Hessian can be modified by Tα, allowing observationally equivalent minima whose Hessian spectral norm is arbitrarily large.This makes spectral-norm sharpness less relevant as a measure of potential generalization error.
- 4.3 ϵ-sharpness: Every rectified-network minimum is observationally equivalent to an equally generalizing minimum with high full-space ϵ-sharpness, while the paper does not establish the same issue for random-subspace ϵ-sharpness.The latter may relate to the wide-valley notion, but the paper reports no corresponding proof.
5 Allowing reparametrizations
Allowing reparametrizations can drastically change loss-surface geometry without changing the represented prediction function or its generalization behavior. The paper demonstrates that sharpness rankings and Hessian-based flatness measures therefore depend on the chosen parameter space.
- Reparametrizing a function can produce arbitrarily different parameter-space geometries without affecting how the function evaluates on unseen data.
- At critical points, reparametrization can make sharp minima correspond to flat minima and conversely.
- Bijections can perturb the relative flatness ranking among a finite number of minima.
- Weight normalization is a practical reparametrization example analyzed because its geometry is simpler.
- For rectifier networks, observationally equivalent parameters allow every minimum to correspond to infinitely sharp or infinitely flat minima under nonzero Hessian eigenvalues.
- Every minimum can also be observationally equivalent to one with arbitrarily low full-space and random-subspace ϵ-sharpness or high full-space ϵ-sharpness.
- Gradient magnitudes of prediction functions are likewise sensitive to input-space reparametrization, including preprocessing such as standardization or whitening.
6 Discussion
Empirical observations that flatter minima generalize better do not hold without qualification across common flatness definitions and deep architectures. Parameter symmetries and reparametrizations can alter flatness without changing the represented function.
- Previously observed flatter minima cannot be assumed to generalize better without further context.
- Model symmetries can alter a minimum’s flatness without affecting the function it represents.
- The geometry of the error surface can change arbitrarily under different parameterizations.
- Flatness cannot be separated from the particular parametrization of the model or input space.
A Radial transformations
The appendix introduces a radial transformation that locally perturbs a finite-dimensional parameter-space geometry. Its effect is confined to a ball around a selected parameter, yet it can change flatness rankings among minima.
- The paper defines an elementary transformation that locally perturbs geometry and changes relative flatness measured by the Hessian spectral norm.
- The radial transformations are parameterized by a center, radius, and local scaling conditions.
- Only the region inside B2(ˆθ, δ) is affected by the transformation.
- Within that local region, the transformations can arbitrarily perturb the flatness ranking of several minima.
B Considering the bias parameter
The bias parameter introduces homogeneity-related constraints in rectified networks. For deeper networks, the resulting condition limits how many Hessian eigenvalues can be arbitrarily influenced.
- For a one-hidden-layer neural network, the bias parameter is analyzed through the non-negative homogeneity property.
- The one-hidden-layer bias analysis yields conclusions similar to those in Section 4.
- For deeper rectified neural networks, the corresponding homogeneity property imposes a condition on the parameter transformations.
- The condition k=1 αk = 1 can decrease the number of Hessian eigenvalues that can be arbitrarily influenced.
C Rectified neural network and Lipschitz continuity
For one-hidden-layer rectified networks, a nonconstant loss is not Lipschitz continuous, and this conclusion extends to deeper rectified architectures.
- A nonconstant loss L for a one-hidden-layer rectified neural network is not Lipschitz continuous.
- The proof also considers the complementary case: a loss with zero gradient almost everywhere must be constant.
- If L has a nonzero gradient at some point, rescaling parameters makes the gradient norm diverge as α approaches 0.
- The non-Lipschitz conclusion generalizes to models containing one-hidden-layer rectified networks, including deeper rectified networks.
D Euclidean distance and input representation
Euclidean metrics can rank models differently after changing input representation, whereas f-divergence and log-likelihood rankings remain invariant under bijective transformations.
- Input representation is critical when ranking models with Euclidean metrics such as mean square error or Earthmover distance.Feature standardization and whitening can significantly change these metrics.
- Bijective transformations do not perturb rankings based on f-divergence and log-likelihood because of the change of variables formula.