Source-linked AI summary
Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
Yusuke Tsuzuku, Issei Sato, Masashi Sugiyama
TL;DR
Existing flatness measures can change when parameters are rescaled, despite generalization being invariant to such rescalings. The paper develops a PAC-Bayesian normalized-flatness measure that accounts for parameter scales. Experiments suggest it distinguishes overfitted models where existing measures can be scale-sensitive.
Problem
Existing flatness definitions are sensitive to parameter rescaling, although generalization is invariant to such rescalings.
Source: §1 · p. 2Method
The paper uses PAC-Bayesian analysis to define normalized flatness that accounts for parameter scales.
Source: §4.1 · p. 6, §8 · p. 15Results
Experiments suggest normalized flatness distinguishes overfitted models even when models are large and existing flatness definitions tend to suffer from scale dependence.
Source: §8 · p. 15Takeaways & Limitations
The authors conclude that normalized flat minima extend flatness analysis beyond cases where parameters are appropriately normalized.
Source: §8 · p. 15The proposed prior and posterior are Gaussian, which the authors note limits model compression relative to more complex posterior structures.
Source: §8 · p. 15
Abstract
from arXiv · showhide
The notion of flat minima has played a key role in the generalization studies of deep learning models. However, existing definitions of the flatness are known to be sensitive to the rescaling of parameters. The issue suggests that the previous definitions of the flatness might not be a good measure of generalization, because generalization is invariant to such rescalings. In this paper, from the PAC-Bayesian perspective, we scrutinize the discussion concerning the flat minima and introduce the notion of normalized flat minima, which is free from the known scale dependence issues. Additionally, we highlight the scale dependence of existing matrix-norm based generalization error bounds similar to the existing flat minima definitions. Our modified notion of the flatness does not suffer from the insufficiency, either, suggesting it might provide better hierarchy in the hypothesis class.
1 Introduction
Flat minima have been linked to generalization, but existing definitions can change under parameter rescaling even when the represented function does not. The paper uses PAC-Bayesian analysis to define scale-invariant flatness and extends the analysis to row- and column-wise rescaling.
1 Introduction
- Existing flatness definitions are scale-dependent, although generalization is invariant to parameter rescaling, leaving open how to combine flatness with parameter scale.Rescaling can change flatness without changing the represented function, including in networks with ReLU or normalization layers.Source: §1 · p. 2
- Earlier PAC-Bayesian analyses used Gaussian priors with the same variance for all parameters, an assumption that does not reflect differing neural-network scales.The paper instead takes scale properties into account.Source: §1 · p. 2
- The analysis incorporates weight-matrix scales and extends to row- and column-wise parameter scaling to define rescaling-invariant flatness.The authors describe this as the first rescaling-invariant definition of flatness for local minima.Source: §1 · p. 2
- Normalized flatness distinguishes models trained on random labels in a setting where a previous definition fails.This is the paper’s stated illustration of the new definition’s utility.Source: §1 · p. 2
2 Related work
Related work connects flat minima and PAC-Bayesian generalization, while prior approaches leave scale dependence unresolved. The paper also identifies scale dependence in matrix-norm bounds and reports a stronger sharpness–generalization-gap correlation for normalized sharpness.
2 Related work
- Prior PAC-Bayesian work noted that sharpness alone is insufficient and proposed scaling it by parameter magnitude, but left the combination open and did not remove scale dependence.Dziugaite & Roy (2017) and Neyshabur et al. (2017) are discussed in this context.Source: §2 · p. 2
- For Wide ResNets trained on CIFAR10 with varying random-label ratios, normalized sharpness has a stronger reported correlation with generalization gap than the trace of the Hessian.The reported correlation coefficients are 91.1 for normalized sharpness and 70.8 for the trace of the Hessian.Source: Figure 1 · p. 3
- The paper says its PAC-Bayesian analysis removes scale dependence and avoids a factor that scales with parameter count, unlike the cited parameter-wise analysis.The comparison concerns Wang et al. (2018).Source: §2 · p. 3
- Prior work normalized convolutional loss landscapes for visualization, while the paper presents PAC-Bayesian theoretical justification for scale normalization.The cited visualization work is Li et al. (2018).Source: §2 · pp. 3–4
3 Flat minima from PAC-Bayesian perspective
The section frames flat minima as parameter-noise stability within a PAC-Bayesian bound. It relates this stability to a bound term, then explains why a Hessian-based sharpness approximation misses prior-variance and KL effects.
3.1 PAC-Bayesian generalization error bound
- The PAC-Bayesian bound controls expected test loss using expected training loss, a KL-divergence term, confidence, and a term depending on the loss and sample size.For 0-1 loss, the paper states that the latter term can be bounded by λ^2/2m.Source: §3.1 · pp. 4–5
3.2 PAC-Bayesian view of flat minima
- Flat minima correspond to term (C), the increase in training loss under parameter noise; when it is sufficiently small, term (A) can also be expected to be small.The analysis focuses on terms (B) and (C).Source: §3.2 · p. 5
3.3 Effect of noises under second-order approximation
- A second-order approximation with unit-variance Gaussian parameter noise approximates term (C) using the Hessian trace.The section connects PAC-Bayes analysis to Hessian-based sharpness measures.Source: §3.3 · pp. 5–6
- The Hessian-trace sharpness measure uses unit variance for every parameter and ignores the KL term, so it does not jointly analyze the two quantities.The paper identifies both issues as central to its subsequent analysis.Source: §3.3 · p. 6
4 Warm-up: Matrix-normalized flat minima
The warm-up section makes PAC-Bayesian prior variance sensitive to each weight matrix’s scale, then defines matrix-normalized sharpness by tuning those variances. The resulting measure is invariant to the specified rescalings and reflects both sharpness and the KL contribution.
4 Warm-up: Matrix-normalized flat minima
- The warm-up motivates scaling the loss landscape by parameter scale because unit-variance Gaussian priors assume all parameters share one scale.The paper notes that scales can differ across weight matrices.Source: §4 · p. 6
4.1 Controlling prior variance
- A zero-mean Gaussian prior and a Gaussian posterior centered at the network parameters are used, with prior and posterior variances for each weight matrix tuned after training.A hyperprior over finite positive floating-point-representable values is used to make prior variance invariant to rescaling.Source: §4.1 · pp. 6–7
4.2 Defining matrix-normalized flat minima
- Matrix-normalized sharpness scales sharpness by each weight matrix’s scale and is invariant to the rescalings proposed by Dinh et al. (2017).It considers both terms (B) and (C) in the PAC-Bayesian bound; with a positive-semidefinite Hessian assumption, the variance choice is used to define the measure.Source: §4.2 · pp. 7–8
5 Normalized flat minima
The section extends flatness analysis to address scale dependence in matrix-wise capacity control, defining normalized flat minima with improved invariance and a reduced effective parameter count in the guarantee’s constant term. It also gives a practical calculation procedure and motivates a surrogate loss whose normalized sharpness better supports model comparison.
5 Normalized flat minima
- Normalized flat minima extend matrix-normalized flatness to row- and column-wise scaling, improving invariance while reducing the effective parameter count in the guarantee’s constant term.The extension is more complicated in form, but still multiplies parameter scales by loss curvatures.Source: §5 · pp. 8–9
5.1 Scale dependence of matrix-wise capacity control
- The transformation preserves the represented ReLU network function while changing Hessian scales, exposing scale dependence in matrix-norm capacity bounds.Under the stated nonzero-row and nonzero-column conditions, both weight matrices can have arbitrarily large spectral and Frobenius norms, with stable ranks arbitrarily close to one.Source: §5.1 · p. 9
5.2 Improving invariance of flatness
- Row- and column-specific hyperprior variances are set as σ_iσ′_j, making the KL constant term O(hd) rather than O(h^2d).The normalized sharpness is defined from the sum of optimization solutions for each weight matrix; in convolutional layers, hyperpriors are needed only on input and output channels.Source: §5.2 · pp. 9–10
5.3 Practical calculation
- The optimization problem for normalized sharpness is convex in α_i and β_i, so near-optimal σ_i and σ′_j can be estimated by gradient descent.The stated convexity also holds for convolutional layers.Source: §5.3 · p. 10
5.4 Choice of the surrogate loss function
- For model comparison, the surrogate loss should keep normalized sharpness unchanged under output shifts or scalings that leave accuracy unchanged.The authors note that cross-entropy after softmax does not meet the scaling condition and use normalized-softmax-cross-entropy in later experiments.Source: §5.4 · pp. 10–11
6 Numerical evaluations
The numerical evaluations test whether normalized sharpness distinguishes models with different generalization behavior and whether normalization addresses scale dependence in common settings. The reported results show larger normalized sharpness for models fitting random labels and a weaker or lost signal from unnormalized sharpness in specified comparisons.
6.1 Distinguishing models trained on random labels
- Networks tended to have larger normalized sharpness when fitting random labels, supporting its use as a hierarchy in the hypothesis class.The study relates normalized sharpness to the train–test accuracy gap across MNIST and CIFAR-10 experiments.Source: §6.1 · p. 12
6.2 Effect of normalization
- Unnormalized sharpness gave a weaker random-label signal and lost its ability to distinguish models in larger networks with normalization layers.The comparison used the same trained models as Section 6.1 and calculated Hessian quantities using cross-entropy; the authors state that scale dependence can be problematic in natural settings.Source: §6.2 · p. 13
7 Discusion
The discussion connects normalized sharpness to MDL and compares its scale-aware design with prior approaches. It also describes how normalized sharpness differs from the Fisher–Rao norm and relates to empirical work on normalization and optimization.
7.1 Connection to MDL arguments
- Normalized sharpness balances posterior variance against the number of bits saved, corresponding to the (B) term in the PAC-Bayesian bound.The discussion frames this balance through the minimum description length principle.Source: §7.1 · p. 13
7.2 Comparison with other prior choices
- The local reparametrization approach has a constant scaling as O(h2d), versus at most O(hd) in this analysis; the cited information-bottleneck discussion is vacuous from a PAC-Bayesian perspective.The paper suggests that its prior design might offer new insights for reinterpreting the information bottleneck.Source: §7.2 · pp. 13–14
- Using initial parameters as the prior mean showed no empirical advantage over zero means in deep networks and adds scale dependence; normalizing that mean is left for future work.Source: §7.2 · p. 14
7.3 Comparison with Fisher-Rao norm
- Normalized sharpness uses Hessian curvature and neural-network parameter structure, unlike the parameter-wise Fisher–Rao norm; under the described rescaling, Fisher–Rao halves while normalized sharpness remains invariant.Matrix-normalized sharpness is also invariant under that rescaling.Source: §7.3 · p. 14
7.4 Supporting empirical findings
- Filter-wise loss-landscape normalization is closely related to row- and column-wise normalization, while cited work links AdamW’s generalization-gap closure with normalized curvature.The paper suggests that normalized flat minima may be useful for rethinking such optimization methods.Source: §7.4 · pp. 14–15
8 Conclusion
The paper proposes normalized flat minima as a definition free from known scale dependence and reports that the analysis can distinguish overfitted models even when they are large. It also identifies the use of Gaussian priors and posteriors as a flaw that limits compression.
8 Conclusion
- The definition is invariant to transformations that affected prior flatness definitions and can approximate generalization bounds tighter than naive parameter counting.Source: §8 · p. 15
- Experimental results suggest the analysis distinguishes overfitted models even when large models and existing flatness definitions face scale-dependence issues.Source: §8 · p. 15
- Using Gaussian prior and posterior distributions may limit model-compression algorithms, since appropriate network posteriors can have more complex structures than Gaussians.The paper identifies richer prior and posterior designs as a direction for future work.Source: §8 · p. 15
A.1 Row and column scaling
The appendix illustrates how row and column rescaling can make matrix norms arbitrarily large, then details the convexity and calculation of normalized sharpness. It contrasts the chosen definition with an alternative that is nonconvex and lacks the same invariance to λ.
A.1 Row and column scaling
- Using the same method, one can make the matrix norms of both W (1) and W (2) arbitrarily large.Source: §A.1 · p. 22
B Convexity of variance parameters
- The optimization in (17) is shown convex with respect to the log of variance parameters, using a positive-semidefinite Hessian argument.The appendix frames the proof by showing the Hessian of the objective is semi-positive definite.Source: B Convexity of variance parameters · p. 23
C Calculation of normalized sharpness
- Calculating normalized sharpness requires solving an optimization problem for each weight matrix, with λ set to 1 for simplicity.The diagonal Hessian elements can be estimated; the small constant r is chosen per matrix according to its Frobenius norm.Source: C Calculation of normalized sharpness · p. 24
D Alternative definition of the normalized sharpness
- The alternative definition (62) avoids a second-order approximation but makes the optimization nonconvex, so the paper uses (17).Unlike (17), (62) does not preserve the property that sharper models remain sharper regardless of λ.Source: D Alternative definition of the… · p. 24
E.1 Setups of Sec. 6.1
The experiments vary the random-label ratio while training MLP, LeNet, and Wide ResNet models with specified datasets and schedules. The Hessian calculation uses the same setup as Sec. E.2 and cross-entropy loss.
E.1 Setups of Sec. 6.1
- Random-label ratios from 0.0 to 1.0 were sampled uniformly at each training, with cross-entropy for training and normalized-softmax-cross-entropy for normalized sharpness.The listed models used Adam at learning rate 0.001 and batch size 128; training ran 50 epochs for MLP and LeNet on MNIST, 100 for LeNet on CIFAR10, and 200 for 16-layer Wide ResNet on CIFAR10 with width factor 4.Source: §E.1 · p. 25
E.2 Setups of Sec. 6.2
- For Sec. 6.2, the experiments use the same setup as Sec. E.2 and cross-entropy loss for both training and Hessian calculation.Source: §E.2 · p. 25