PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization

Nour Jamoussi, Marios Kountouris

arXiv:2609.09367v1cs.LGcs.IT

TL;DR

The paper asks how divergence-based regularization relates locally to SAM, whose shared connection to generalization is otherwise not fully characterized. It derives a second-order geometric framework for parameter and input perturbations and uses asymmetric α-skew JSD as a controlled test case. The analysis identifies Fisher and pullback quadratic forms, with maximal α(1−α) curvature at α = 1/2 associated with improved performance and flatter local landscapes.

  • Problem

    The local second-order relationship between divergence-based regularization and SAM has not been fully characterized.

  • Method

    The paper derives a unified local second-order expansion for f-divergences in parameter and input spaces, relating the resulting geometry to SAM and testing α-skew JSD.

  • Results

    Within the α-skew JSD family, local curvature scales as α(1−α), peaks at α = 1/2, and stronger curvature is associated with improved performance and flatter input-perturbation landscapes.

  • Takeaways & Limitations

    Divergence-based regularization provides a local sensitivity framework that connects Fisher-weighted parameter perturbations with pullback-metric input perturbations.

  • Takeaways & Limitations

    Second-order analysis cannot determine which divergence family should be preferred because all f-divergences coincide locally up to a scaling factor.

Abstract

from arXiv · show

Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning, both motivated by robustness to perturbations. However, their relationship has remained largely unexplored. Building on classical second-order expansions of $f$-divergences, we show that the two methods are locally consistent under parameter-space perturbations: both induce curvature-sensitive penalties, with divergence regularization yielding a Fisher-weighted quadratic form and SAM penalizing sharpness through the dominant Hessian eigenvalue. For negative log-likelihood objectives with exponential-family output distributions, this correspondence becomes especially transparent, since the Fisher and Gauss-Newton matrices coincide. We further show that the same local geometric perspective extends to input-space perturbations, where divergence-based regularization is defined through transformations of the input. In this setting, the regularizer induces a pullback quadratic form on the input space, providing a more general perturbation framework than standard SAM while preserving the same local sensitivity interpretation. To validate the analysis empirically, we use the asymmetric $α$-skew Jensen-Shannon divergence (JSD) family as a controlled testbed. Its local curvature coefficient scales as $α(1-α)$ and is maximized at the symmetric point $α=\tfrac12$, which recovers the standard JSD. Loss-landscape visualizations in the input-perturbation regime show that stronger induced curvature penalization is associated with flatter local minima. Experiments on four benchmark datasets further demonstrate that both accuracy and negative log-likelihood are consistently best near this regime of maximal curvature penalization.

1 INTRODUCTION

The paper studies whether divergence-based regularization and SAM share a local second-order geometric basis for improving generalization. It develops a unified curvature view spanning parameter and input perturbations, then tests it with asymmetric α-skew JSD regularization.

  • Generalization requires reliable predictions on unseen examples and across changing observations, tasks, environments, and small distributional shifts.
  • The paper addresses the unresolved local-geometric relationship between divergence-based regularization and SAM.
  • Small perturbations induce quadratic penalties described by Fisher geometry in parameter space and a pullback metric in input space.
  • Under standard probabilistic-loss assumptions, parameter-space divergence geometry is locally comparable to second-order interpretations of SAM.
  • The asymmetric α-skew JSD family provides a controlled test of how induced curvature relates to predictive performance and local landscape flatness.

2 RELATED WORK

Related work develops two complementary approaches to robustness: output-distribution consistency under perturbations and parameter-space sharpness-aware optimization. This paper connects them by deriving the local geometry induced by general f-divergence regularization.

  • Divergence-based regularization and information-geometric penalties: Divergence-based methods penalize discrepancies between predictive distributions under perturbations, encouraging output consistency and distributional smoothness.
  • Sharpness-aware minimization: SAM minimizes worst-case loss in parameter neighborhoods, favoring flat regions of the loss landscape.
  • The paper starts from f-divergence regularization and derives its locally induced geometry, rather than modifying SAM’s perturbation geometry.

3 A GENERAL FRAMEWORK FOR f-DIVERGENCE REGULARIZATION

The framework models divergence regularization under input, parameter, or combined perturbations and analyzes its local quadratic behavior. Near matching distributions, all f-divergences share a scaled quadratic form that becomes Fisher-weighted in parameter space and pullback-based in input space, with α-skew JSD providing a curvature-controlled example.

  • General setup: The empirical loss approximates the population loss because the data distribution is unknown, but direct minimization can overfit and motivates regularization.
  • General setup: Divergence regularization compares the model’s original output distribution with outputs produced by input transformations, parameter perturbations, or both.
  • Local quadratic geometry: Near identical distributions, every f-divergence reduces to a quadratic penalty on predictive-distribution changes, differing locally only through ϕ′′(1).
  • Local quadratic geometry: Parameter perturbations yield a Fisher-information form, while input perturbations yield a pullback metric and thereby penalize local predictive sensitivity.
  • Curvature analysis: The asymmetric α-skew JSD family offers a controlled one-parameter comparison of induced curvature within a fixed divergence family.
  • Curvature analysis: Within this family, the local curvature coefficient scales with α(1−α) and is greatest at the symmetric standard-JSD point α = 1/2.

4 LOCAL CONNECTION TO SHARPNESS-AWARE MINIMIZATION

The paper compares divergence regularization and SAM through local second-order curvature. Divergence regularization yields Fisher-weighted penalties, while SAM controls dominant Hessian curvature, with closer alignment under Gauss-Newton conditions.

  • SAM curvature interpretation: Near stationarity with nonnegative dominant curvature, SAM locally penalizes sharpness through the dominant Hessian eigenvalue.This follows by analyzing the inner maximization with a second-order Taylor expansion and the Rayleigh-Ritz characterization.
  • Local comparison: Divergence regularization imposes a Fisher-weighted quadratic penalty on parameter perturbations, whereas SAM imposes a spectral curvature penalty.The two penalties therefore use different local geometric summaries of parameter sensitivity.
  • Curvature alignment: For NLL objectives with exponential-family outputs, the Fisher and generalized Gauss-Newton matrices coincide, aligning divergence-induced geometry with SAM’s second-order curvature structure.Softmax outputs trained with cross-entropy are included in this setting.
  • Input-space extension: Divergence regularization extends the same local sensitivity framework to input or combined input-parameter perturbations, beyond SAM’s parameter-only regime.In input space, the resulting penalty targets directions where predictive distributions vary sharply under small input changes.

5 EMPIRICAL VALIDATION

The empirical study varies the asymmetry parameter of the α-skew JSD family while evaluating input-perturbation regularization across four datasets. Performance and loss-landscape results are strongest near α = 1/2, where the induced curvature coefficient is maximal.

  • 5.1 EXPERIMENTAL SETUP: The input-perturbation experiments use 2% random masking to compare predictive distributions on original and transformed inputs.Accuracy and NLL provide complementary measures of correctness and probabilistic prediction quality.
  • 5.1 EXPERIMENTAL SETUP: The α-skew JSD experiments vary only α ∈ {0.1, 0.2, ..., 0.9}, isolating curvature changes while keeping the training pipeline fixed.The local second-order coefficient is α(1 − α), which is maximal at the symmetric setting.
  • 5.2 EMPIRICAL PERFORMANCE ACROSS THE ASYMMETRIC α-SKEW JSD FAMILY: Across four benchmark datasets, accuracy tends to increase and NLL decrease as α approaches 1/2, with best overall results at α = 1/2.The evaluated datasets are CIFAR-10, Fashion-MNIST, EMNIST (Balanced), and Oxford-IIIT Pet.
  • 5.3 LOSS LANDSCAPE COMPARISON: Relative to the baseline, α = 0.9 produces a flatter loss landscape, while α = 0.5 produces the flattest basin among the compared models.The visualization compares the baseline, α = 0.9, and α = 0.5 on a common perturbation plane with shared loss colors.
  • 5.3 LOSS LANDSCAPE COMPARISON: Across three seeds, divergence regularization lowers Hessian-trace and maximum-eigenvalue measures relative to baseline, with α = 0.5 showing the smallest curvature measures.The same setting also yields the lowest average loss, although seed variability is non-negligible.

6 LIMITATIONS

Second-order analysis gives all f-divergences the same local quadratic form up to a scaling factor. Consequently, it cannot by itself select a preferred divergence family.

  • 6 LIMITATIONS: Second-order analysis cannot determine which f-divergence family should be preferred because all share the same local quadratic form up to ϕ′′(1).Meaningful differences may instead arise from higher-order terms or structural properties such as symmetry and boundedness.

7 CONCLUSION

The paper establishes a local geometric connection between f-divergence regularization and SAM, then extends the perspective to input perturbations. In the asymmetric α-skew JSD family, maximal local curvature near α = 1/2 is associated with improved performance and flatter input-perturbation landscapes.

  • 7 CONCLUSION: f-divergence regularization is locally comparable to second-order interpretations of SAM in parameter space and extends naturally to input perturbations.The input-space formulation defines a more general perturbation regime than SAM.
  • 7 CONCLUSION: The asymmetric α-skew JSD coefficient scales as α(1 − α) and is maximized at the symmetric point α = 1/2.Within this controlled family, stronger coefficients were associated with improved performance and flatter projected loss landscapes.

A PROOFS

The proofs establish local quadratic forms for parameter and input perturbations and characterize quadratic maximization over a Euclidean ball through the largest eigenvalue.

  • A.1 FISHER FORM: The Fisher-form proof expands the predictive distributions under a small parameter perturbation and identifies the resulting matrix as the Fisher information matrix.The derivation concludes after expressing the differential in coordinates and applying the Fisher-matrix definition.
  • A.2 PULLBACK FORM: The pullback-form proof considers a small input transformation T(x) = x + ∆x and differentiates the predictive distribution with respect to the input.The supplied proof passages state the setup but do not include the complete resulting quadratic expression.
  • A.3 ASYMMETRIC α-SKEW JSD: The asymmetric α-skew JSD expansion is derived by decomposing the two KL terms around Mα = (1 − α)P + αQ and using a Padé approximation with Taylor expansion.The proof separately expands DKL(P||Mα) and DKL(Q||Mα) before combining them to obtain the second-order result.
  • A.4 QUADRATIC MAXIMIZATION OVER THE EUCLIDEAN BALL: For nonnegative dominant curvature, maximizing a symmetric quadratic form over a Euclidean ball occurs on the boundary along an eigenvector of the largest eigenvalue.This follows from the Rayleigh quotient characterization.

B ILLUSTRATIVE BEHAVIOR OF THE ASYMMETRIC α-SKEW JSD FAMILY ON UNIVARIATE GAUSSIANS

For two separated univariate Gaussians, the asymmetric α-skew JSD interpolates through mixture distributions as α varies. The divergence approaches zero at the endpoints and is largest at the symmetric point α = 1/2.

  • B ILLUSTRATIVE BEHAVIOR OF THE ASYMMETRIC α-SKEW JSD FAMILY ON UNIVARIATE GAUSSIANS: The mixture Mα transitions smoothly from P to Q as α increases, while JSDα reaches its maximum at α = 1/2.The illustrated distributions are P = N(−1, 1) and Q = N(1, 1).
  • B ILLUSTRATIVE BEHAVIOR OF THE ASYMMETRIC α-SKEW JSD FAMILY ON UNIVARIATE GAUSSIANS: JSDα approaches zero at the endpoints of the α range, corresponding to the asymmetric mixture becoming concentrated at one distribution.The supplied passage explicitly connects the maximum to the symmetric JSD but does not further quantify the endpoint behavior.

C DETAILED DESCRIPTION OF THE TRANSFORMATION USED

The input transformation randomly masks a small subset of image pixels using an independently sampled spatial map. Selected pixels are replaced by the corresponding image’s mean intensity.

  • C DETAILED DESCRIPTION OF THE TRANSFORMATION USED: Selected pixels are replaced with a baseline equal to the mean intensity of the corresponding image.

D.1 SYMMETRIC POWER-LAW CHARACTERIZATION OF THE EMPIRICAL CURVES

The paper fits symmetric power-law curves to characterize how empirical metrics vary with α around the symmetric point. Across datasets, the fitted curvature reveals sharper, smoother, or nearly quadratic responses, while loss landscapes become flatter under divergence regularization.

  • Symmetric power-law characterization: CIFAR-10 shows sharper performance gains near α = 0.5, with fitted curves exhibiting pronounced concave and convex shapes and p < 2.The exponent p measures the effective curvature of the empirical response; smaller p indicates sharper central variation.
  • Symmetric power-law characterization: EMNIST exhibits a smoother trade-off across α, with fitted power-law exponents between 3 and 4.Higher p values correspond to milder variation around the symmetric point in this empirical characterization.
  • Symmetric power-law characterization: Fashion-MNIST and Oxford-IIIT Pet follow nearly quadratic, symmetric trends with p approximately equal to 2.These trends reinforce the stability of the symmetric divergence regime in the fitted empirical curves.
  • Loss-landscape visualization: Divergence-based regularization leads to progressively flatter local minima than the unregularized baseline in shared-scale loss-landscape slices.The comparison uses models trained with α = 0.9 and α = 0.5 asymmetric α-skew JSD regularization.
Loading 2609.09367v1…