Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization

• Generalization requires reliable predictions on unseen examples and across changing observations, tasks, environments, and small distributional shifts.Source: §1 · p. 1
• The paper addresses the unresolved local-geometric relationship between divergence-based regularization and SAM.Source: §1 · p. 2
• Small perturbations induce quadratic penalties described by Fisher geometry in parameter space and a pullback metric in input space.Source: §1 · p. 2
• Under standard probabilistic-loss assumptions, parameter-space divergence geometry is locally comparable to second-order interpretations of SAM.Source: §1 · p. 2
• The asymmetric α-skew JSD family provides a controlled test of how induced curvature relates to predictive performance and local landscape flatness.Source: §1 · p. 2

Divergence-based regularization and information-geometric penalties

• Divergence-based methods penalize discrepancies between predictive distributions under perturbations, encouraging output consistency and distributional smoothness.Source: §2 · p. 2

Sharpness-aware minimization

• SAM minimizes worst-case loss in parameter neighborhoods, favoring flat regions of the loss landscape.Source: §2 · p. 2

• The paper starts from f-divergence regularization and derives its locally induced geometry, rather than modifying SAM’s perturbation geometry.Source: §2 · p. 2

General setup

• The empirical loss approximates the population loss because the data distribution is unknown, but direct minimization can overfit and motivates regularization.Source: §3 · p. 3
• Divergence regularization compares the model’s original output distribution with outputs produced by input transformations, parameter perturbations, or both.Source: §3 · p. 3

Local quadratic geometry

• Near identical distributions, every f-divergence reduces to a quadratic penalty on predictive-distribution changes, differing locally only through ϕ′′(1).Source: §3.1 · p. 4
• Parameter perturbations yield a Fisher-information form, while input perturbations yield a pullback metric and thereby penalize local predictive sensitivity.Source: §3.1 · p. 4

Curvature analysis

• The asymmetric α-skew JSD family offers a controlled one-parameter comparison of induced curvature within a fixed divergence family.Source: §3.2 · p. 5
• Within this family, the local curvature coefficient scales with α(1−α) and is greatest at the symmetric standard-JSD point α = 1/2.Source: §3.2 · p. 5

SAM curvature interpretation

• Near stationarity with nonnegative dominant curvature, SAM locally penalizes sharpness through the dominant Hessian eigenvalue.This follows by analyzing the inner maximization with a second-order Taylor expansion and the Rayleigh-Ritz characterization.Source: §4 · p. 6

Local comparison

• Divergence regularization imposes a Fisher-weighted quadratic penalty on parameter perturbations, whereas SAM imposes a spectral curvature penalty.The two penalties therefore use different local geometric summaries of parameter sensitivity.Source: §4 · pp. 5–6

Curvature alignment

• For NLL objectives with exponential-family outputs, the Fisher and generalized Gauss-Newton matrices coincide, aligning divergence-induced geometry with SAM’s second-order curvature structure.Softmax outputs trained with cross-entropy are included in this setting.Source: §4 · pp. 6–7

Input-space extension

• Divergence regularization extends the same local sensitivity framework to input or combined input-parameter perturbations, beyond SAM’s parameter-only regime.In input space, the resulting penalty targets directions where predictive distributions vary sharply under small input changes.Source: §4 · p. 7

5.1 EXPERIMENTAL SETUP

• The input-perturbation experiments use 2% random masking to compare predictive distributions on original and transformed inputs.Accuracy and NLL provide complementary measures of correctness and probabilistic prediction quality.Source: §5.1 · p. 7
• The α-skew JSD experiments vary only α ∈ {0.1, 0.2, ..., 0.9}, isolating curvature changes while keeping the training pipeline fixed.The local second-order coefficient is α(1 − α), which is maximal at the symmetric setting.Source: §5.1 · p. 7, §5 · p. 7

5.2 EMPIRICAL PERFORMANCE ACROSS THE ASYMMETRIC α-SKEW JSD FAMILY

• Across four benchmark datasets, accuracy tends to increase and NLL decrease as α approaches 1/2, with best overall results at α = 1/2.The evaluated datasets are CIFAR-10, Fashion-MNIST, EMNIST (Balanced), and Oxford-IIIT Pet.Source: §5.1 · p. 7, §5.2 · p. 8, Figure 1 · p. 8

5.3 LOSS LANDSCAPE COMPARISON

• Relative to the baseline, α = 0.9 produces a flatter loss landscape, while α = 0.5 produces the flattest basin among the compared models.The visualization compares the baseline, α = 0.9, and α = 0.5 on a common perturbation plane with shared loss colors.Source: §5.3 · pp. 8–9, Figure 2 · p. 9
• Across three seeds, divergence regularization lowers Hessian-trace and maximum-eigenvalue measures relative to baseline, with α = 0.5 showing the smallest curvature measures.The same setting also yields the lowest average loss, although seed variability is non-negligible.Source: §5.3 · p. 9, Table 1 · p. 9

6 LIMITATIONS

• Second-order analysis cannot determine which f-divergence family should be preferred because all share the same local quadratic form up to ϕ′′(1).Meaningful differences may instead arise from higher-order terms or structural properties such as symmetry and boundedness.Source: §6 · p. 9

7 CONCLUSION

• f-divergence regularization is locally comparable to second-order interpretations of SAM in parameter space and extends naturally to input perturbations.The input-space formulation defines a more general perturbation regime than SAM.Source: §7 · p. 10
• The asymmetric α-skew JSD coefficient scales as α(1 − α) and is maximized at the symmetric point α = 1/2.Within this controlled family, stronger coefficients were associated with improved performance and flatter projected loss landscapes.Source: §7 · p. 10

A.1 FISHER FORM

• The Fisher-form proof expands the predictive distributions under a small parameter perturbation and identifies the resulting matrix as the Fisher information matrix.The derivation concludes after expressing the differential in coordinates and applying the Fisher-matrix definition.Source: §A.1 · p. 12

A.2 PULLBACK FORM

• The pullback-form proof considers a small input transformation T(x) = x + ∆x and differentiates the predictive distribution with respect to the input.The supplied proof passages state the setup but do not include the complete resulting quadratic expression.Source: §A.2 · p. 12

A.3 ASYMMETRIC α-SKEW JSD

• The asymmetric α-skew JSD expansion is derived by decomposing the two KL terms around Mα = (1 − α)P + αQ and using a Padé approximation with Taylor expansion.The proof separately expands DKL(P||Mα) and DKL(Q||Mα) before combining them to obtain the second-order result.Source: §A.3 · p. 13

A.4 QUADRATIC MAXIMIZATION OVER THE EUCLIDEAN BALL

• For nonnegative dominant curvature, maximizing a symmetric quadratic form over a Euclidean ball occurs on the boundary along an eigenvector of the largest eigenvalue.This follows from the Rayleigh quotient characterization.Source: §A.4 · p. 15

B ILLUSTRATIVE BEHAVIOR OF THE ASYMMETRIC α-SKEW JSD FAMILY ON UNIVARIATE GAUSSIANS

• The mixture Mα transitions smoothly from P to Q as α increases, while JSDα reaches its maximum at α = 1/2.The illustrated distributions are P = N(−1, 1) and Q = N(1, 1).Source: B ILLUSTRATIVE BEHAVIOR OF THE… · p. 16, Figure 3 · p. 16
• JSDα approaches zero at the endpoints of the α range, corresponding to the asymmetric mixture becoming concentrated at one distribution.The supplied passage explicitly connects the maximum to the symmetric JSD but does not further quantify the endpoint behavior.Source: B ILLUSTRATIVE BEHAVIOR OF THE… · p. 16

C DETAILED DESCRIPTION OF THE TRANSFORMATION USED

• Selected pixels are replaced with a baseline equal to the mean intensity of the corresponding image.Source: C DETAILED DESCRIPTION OF THE… · p. 16

Symmetric power-law characterization

• CIFAR-10 shows sharper performance gains near α = 0.5, with fitted curves exhibiting pronounced concave and convex shapes and p < 2.The exponent p measures the effective curvature of the empirical response; smaller p indicates sharper central variation.Source: §D.1 · p. 17, Figure 5 · p. 17
• EMNIST exhibits a smoother trade-off across α, with fitted power-law exponents between 3 and 4.Higher p values correspond to milder variation around the symmetric point in this empirical characterization.Source: §D.1 · p. 17
• Fashion-MNIST and Oxford-IIIT Pet follow nearly quadratic, symmetric trends with p approximately equal to 2.These trends reinforce the stability of the symmetric divergence regime in the fitted empirical curves.Source: §D.1 · p. 17

Loss-landscape visualization

• Divergence-based regularization leads to progressively flatter local minima than the unregularized baseline in shared-scale loss-landscape slices.The comparison uses models trained with α = 0.9 and α = 0.5 asymmetric α-skew JSD regularization.Source: §D.2 · p. 18, Figure 6 · p. 18