Curvature-Scaled Sharpness-Aware Minimization: Convergence Beyond Smoothness

1 Introduction

• Generalized smoothness models allow curvature bounds to grow with gradient norm or loss, and have theoretical support for specific neural-network settings.The introduction identifies this growth as motivation for gradient-dependent and loss-dependent curvature bounds.Source: §1 · p. 2
• SAM's fixed-radius perturbation can impede exact convergence, while gradient-proportional perturbations can become too large at high gradients under (L0, L1)-smoothness.The introduction frames controlling perturbation effects at both low and high gradient scales as the challenge for SAM.Source: §1 · p. 2

1.1 Main Contributions

• Under (L0, L1)-smoothness, SAM's distortion can become unbounded near stationarity, while USAM's can become unbounded at large gradients; examples include cycling and divergence.A necessary condition for uniform distortion control requires gradient-aligned perturbations to be at most proportional to the gradient norm near stationarity and bounded at large gradients.Source: §1.1 · pp. 2–3
• CS-SAM scales its perturbation and descent step by a curvature scale, and admissible scales yield distortion bounded by e^ρ − 1 at every nonstationary iterate.Both gradient-dependent and loss-dependent curvature scales satisfy the stated conditions.Source: §1.1 · p. 3
• Under both generalized-smoothness models, CS-SAM has O(1/T) rates for the best squared gradient norm on nonconvex objectives and last-iterate function gap on convex objectives with a minimizer, plus linear convergence under PL.With ρ = 0, the guarantees recover gradient-descent rates for curvature-scaled steps.Source: §1.1 · p. 3
• The loss-dependent CS-SAM variant achieves the highest mean best-test accuracy in all four settings, while the gradient-dependent variant exceeds tuned SAM and USAM baselines in three of four.The comparisons include constant-step and cosine-annealing schedules.Source: §1.1 · p. 3

2 From gradient distortion to curvature-scaled SAM

• The section frames relative gradient distortion as the error between the perturbed-point gradient and the gradient used by ordinary descent.The section defines this quantity and uses it to motivate the CS-SAM update and its uniform distortion guarantee.Source: §2 · p. 3, §2.1 · p. 4

2.1 Relative gradient distortion and the failures of SAM and USAM

• SAM’s distortion can become unbounded as the gradient vanishes, whereas USAM’s can grow unboundedly as the gradient grows.On one-dimensional convex objectives, SAM admits a nonzero two-cycle on a quadratic, while USAM diverges on a quartic from sufficiently large initializations.Source: §2.1 · pp. 4–5
• Uniform distortion control requires gradient-collinear perturbation lengths to grow at most linearly near zero gradient and remain bounded at large gradients.This condition makes the perturbation USAM-like near stationarity and SAM-like for large gradients; the proposed multiplier ρ/(L0 + L1s) satisfies it.Source: §2.1 · p. 5

2.2 Admissible scales and the CS-SAM update

• An admissible scale upper-bounds local curvature, varies at most exponentially with distance, and keeps a curvature-scaled gradient step short along the perturbation.The gradient-dependent scale ML is L1-admissible under (L0, L1)-smoothness; the constant and loss-dependent scales are also admissible under their respective models.Source: §2.2 · p. 6
• CS-SAM scales both its perturbation and descent steps by an admissible curvature scale, evaluated at the point where each corresponding gradient is taken.At ρ = 0, the update reduces to curvature-scaled gradient descent with step γ/M(xt).Source: §2.2 · p. 6
• CS-SAM uniformly bounds relative gradient distortion by e^ρ −1 for any admissible curvature scale, without dependence on the scale or its admissibility constant.The distortion bound is below 1 when ρ < log 2, a necessary condition for the convergence rates.Source: §2.2 · p. 7

3 Convergence rates

• The rate conditions restrict perturbation and step sizes; convex guarantees additionally require c0 ≥ 0.χ(ρ, γ) > 0 is required throughout, while c0 ≥ 0 is required for convex results.Source: §3 · p. 7

3.1 (L0, L1) curvature scale: M = ML

• For M = ML, the best squared gradient norm is O(1/T) on nonconvex objectives, the last-iterate convex gap is O(1/T), and convergence is linear under PL.The nonconvex rate requires χ(ρ, γ) > 0; the convex rate additionally requires c0 ≥ 0. No bounded-gradient or bounded-iterate assumption is needed for the nonconvex result.Source: §3.1 · p. 8
• At ρ = 0, the ML method reduces to gradient descent with step γ/(L0 + L1∥∇f(xt)∥), recovering its known asymptotic rate orders.The paper identifies these rates with those analyzed by Vankov et al. [2025] and Gorbunov et al. [2025], and extends them to positive perturbations.Source: §3.1 · p. 8

3.2 (H0, H1) curvature scale: M = MH

• For M = MH, the best squared gradient norm and convex last-iterate gap are O(1/T), and convergence is linear under PL.The H1 contribution appears in the leading 1/T term; the PL contraction factor is 1 − 2µγe^−ρ χ(ρ, γ)/(H0 + H1(f(x0) − f⋆)).Source: §3.2 · p. 9
• At ρ = 0, the MH method reduces to gradient descent with step γ/(H0 + H1(f(xt) − f⋆)); its guarantees match known rate orders and permit larger step sizes.The loss-dependent step uses f⋆, unlike the gradient-dependent scale; the two scales give guarantees for different step choices.Source: §3.2 · p. 9

4 Numerical experiments

• The deterministic verification uses one-dimensional quadratic and quartic objectives; the DNN experiments use minibatch gradients and evaluate curvature scales on minibatch quantities.The DNN settings train ResNet-20/32 on CIFAR-10/100 for 200 epochs with batch size 128.Source: §4.1 · p. 10, §4.2 · pp. 10–11

4.1 Verification of Section 2

• On the quadratic, SAM approaches a nonzero two-cycle; on the quartic, USAM diverges from the stated initialization, while CS-SAM converges on both objectives with either scale.For the quartic USAM run, |x2| reaches approximately 10^11 after two updates.Source: §4.1 · p. 10
• SAM distortion grows unbounded as the gradient vanishes, and USAM distortion grows unbounded as the gradient grows; CS-SAM stays below e^ρ − 1 for both scales.On the quadratic CS-SAM distortion equals ρ; on the quartic it peaks below the bound and decays at both ends.Source: §4.1 · p. 10

4.2 DNN experiments

• CS-SAM with the MH scale has the highest mean best-test accuracy in all four settings, while ML beats tuned SAM and USAM in three of four.The comparisons use tuned SAM and USAM with constant and cosine-annealing schedules.Source: §4.2 · p. 11, Table 3 · p. 11

Conclusion

• SAM's distortion can become unbounded near stationarity, while USAM's can become unbounded at large gradients, with examples of SAM cycling and USAM diverging.The paper also establishes a necessary condition for uniform distortion control among gradient-direction perturbations whose nonnegative multipliers depend only on gradient norm.Source: §5 · p. 11
• CS-SAM scales both the perturbation and descent step by admissible curvature upper bounds, uniformly bounding relative gradient distortion.Its scaling is motivated by the necessary condition for uniform distortion control.Source: §5 · p. 11
• Under both (L0, L1)- and (H0, H1)-smoothness, CS-SAM achieves deterministic O(1/T) best squared-gradient rates for nonconvex objectives and last-iterate O(1/T) function-gap rates for convex objectives.The paper also proves linear convergence under the Polyak-Lojasiewicz condition.Source: §5 · p. 11
• At zero perturbation, the guarantees recover gradient-descent rates with (L0, L1)- or (H0, H1)-based step-sizes.The experiments confirm the failures and distortion bound on one-dimensional objectives, and show both curvature scales competitive with tuned SAM and USAM on ResNet-20/32 and CIFAR-10/100.Source: §5 · p. 11
• The minibatch vision experiments report that the (H0, H1) variant has the highest mean best-test accuracy in all four settings against tuned SAM and USAM with constant and cosine annealing schedules.Future directions include stochastic analysis, removing the loss scale's dependence on f⋆, and combining CS-SAM with momentum and weight decay in larger-scale training.Source: §5 · p. 11

AI use statement

• Generative AI tools suggested proof ideas, which the authors checked and wrote up.The authors state that they take responsibility for the final content.Source: AI use statement · p. 11
• The authors used generative AI tools to streamline and edit the wording of some paragraphs and sentences.They state that they take responsibility for the final content.Source: AI use statement · p. 11
• The authors take responsibility for the final content of the work.They also report using AI tools for suggested proof ideas and wording edits.Source: AI use statement · p. 11

SAM, normalization, and optimization dynamics

• Fixed-radius SAM has been linked to two-cycles, stationarity neighborhoods, and perturbed-gradient stationary points [Bartlett et al., 2023; Si and Yun, 2023; Park et al., 2026], while this paper identifies complementary low-gradient SAM and high-gradient USAM distortion failures.The paper frames these as consequences of expressing the perturbation budget in inappropriate units.Source: A Further Related Work · p. 16

Unified and adaptive perturbation rules

• Unlike adaptive SAM rules based on validation, optimizer statistics, parameter scale, blockwise gradient energy, or loss-space budgets, CS-SAM derives its perturbation scaling from uniform distortion control and uses the same curvature scale for the outer step.The related-work discussion contrasts CS-SAM with Unified SAM [Oikonomou and Loizou, 2025], ASAM [Kwon et al., 2021], Fisher SAM [Kim et al., 2022], LETS [Wang et al., 2024], LightSAM [Cheng et al., 2025], and LE-SAM [Wang et al., 2026].Source: A Further Related Work · p. 16

Relative-error control and exact stationarity

• Relative error control is established prior work [Khanh et al., 2024; Adly and Le, 2026], but this paper derives the scaling required for uniform control under gradient-dependent curvature.The derived rule is USAM-like near stationarity and SAM-like at large gradients, and couples the perturbation to a curvature-scaled outer step.Source: A Further Related Work · pp. 16–17

Optimization beyond uniform smoothness

• Curvature-scaled gradient steps are established machinery, while CS-SAM applies the curvature scale to the perturbation too and obtains the same asymptotic orders in T as curvature-scaled gradient descent.The rates have different dependence on curvature parameters and initial conditions, and recover gradient-descent rates at ρ = 0.Source: A Further Related Work · p. 17

Relative gradient models and curvature geometry

• The analysis specializes relative-locality ideas to a first-order sharpness-aware perturbation and certifies a two-gradient SAM update on nonconvex objectives, unlike convex Newton-oriented generalized self-concordance.Source: A Further Related Work · p. 17

Geometry, generalization, and scope

• CS-SAM is presented as an optimization method, not a parameterization-invariant theory of generalization; the analysis is deterministic, assumes exact gradients and function values, and uses a global Euclidean scalar curvature scale.Parameter-space sharpness is sensitive to neural-network reparameterization [Andriushchenko et al., 2023; Kwon et al., 2021].Source: A Further Related Work · p. 17

Auxiliary framework

• The appendix states most auxiliary lemmas for a general κ-admissible curvature scale, with separate results for the loss scale and elementary sequence bounds.It also includes the differential form of Grönwall’s inequality.Source: B Auxiliary lemmas · p. 18

Gradient-gap bound

• For a κ-admissible scale, the gradient gap is bounded below by (2 log 2 − 1) times the squared gradient norm divided by the scale.The coefficient is maximized at α = log 2.Source: B Auxiliary lemmas · p. 19

Perturbed-gradient bounds

• For CS-SAM iterates, the perturbation bounds curvature-scale variation and gradient change, and controls the perturbed gradient norm and its alignment with the original gradient.The stated bounds include M(yt) between e^-ρ M(xt) and e^ρ M(xt), and an inner product lower bound of (2 − e^ρ) times the squared original gradient norm.Source: B Auxiliary lemmas · p. 19

Descent bound

• When χ(ρ, γ) > 0, the CS-SAM step decreases the objective by at least γe^-ρ χ(ρ, γ) times the squared gradient norm divided by the curvature scale.The descent bound is derived using the perturbed-gradient alignment and scale bounds.Source: B Auxiliary lemmas · pp. 18–20

Convex-objective bound

• For convex objectives with a minimizer, the auxiliary argument expands squared distance to the minimizer and uses convexity and the gradient-gap bound to control the function gap.The proof uses the CS-SAM perturbation and perturbed-gradient estimates.Source: B Auxiliary lemmas · p. 20

Loss scale and rate lemma

• For the loss scale, the appendix derives a gradient bound using the loss gap, and an elementary sequence lemma turns quadratic gap decrease into a 1/T rate.If the loss scale vanishes at one point, the stated result is that it vanishes everywhere and the objective is constant.Source: B Auxiliary lemmas · pp. 20–21

C Proofs for Section 2

• For f(x)=x^2/2, SAM has a nonzero two-cycle at ±a, and no trajectory converges to the minimizer unless it lands there exactly.The cycle amplitude is a=ηr/(2−η).Source: C Proofs for Section 2 · p. 23
• For f(x)=x^4/4, USAM diverges with alternating signs from initializations above the stated threshold.The iterates’ absolute values increase strictly and tend to infinity.Source: C Proofs for Section 2 · p. 23
• Uniform distortion at most δ forces α(s)≤δ/L0 and α(s)≤log(1+δ)/(L1s) for every gradient norm s>0.These bounds constrain perturbation length both near stationarity and as the gradient grows.Source: C Proofs for Section 2 · p. 24
• Under (L0,L1)-smoothness, ML is admissible with κ=L1; under (H0,H1)-smoothness, positive MH is admissible with κ=√H1.The proof also notes that if MH vanishes somewhere, then f is constant.Source: C Proofs for Section 2 · pp. 24–25
• Theorem 2.5’s proof obtains CS-SAM’s distortion bound by applying a lemma valid for every κ-admissible scale.The argument divides the lemma’s second inequality by the current gradient norm.Source: C Proofs for Section 2 · p. 25

D Proofs for Section 3

• Appendix D.1 and D.2 provide the proofs of Theorems 3.1 and 3.2, respectively.Source: D Proofs for Section 3 · p. 26

• Theorem 3.1 gives a best-squared-gradient-norm rate for nonconvex objectives, proved by summing descent over the first T iterations.The proof uses that s^2/(L0+L1s) is increasing to relate the sum to the smallest gradient norm.Source: §D.1 · p. 26
• For convex objectives with a minimizer and c0≥0, Theorem 3.1 gives a last-iterate function-gap rate.The proof uses nonincreasing distance to the minimizer and convexity.Source: §D.1 · pp. 26–27
• Under the µ-PL condition, Theorem 3.1 gives a PL rate by showing the function-gap contraction factor lies in [0,1).The proof obtains the factor using the PL inequality and the monotonicity of s^2/(L0+L1s).Source: §D.1 · pp. 26–27

D.2 Proofs for Section 3.2: the (H0, H1) curvature scale

• The H1 contribution enters Theorem 3.2’s rates at leading order, unlike in Theorem 3.1.The proof attributes this difference to the constant coefficient on ∥∇f(xt)∥^2 in its descent inequality.Source: §D.2 · p. 27
• Theorem 3.2 gives a best-squared-gradient-norm rate for nonconvex objectives by summing descent over T iterations.The proof bounds the minimum gradient norm by the average across those iterations.Source: §D.2 · pp. 27–28
• For convex objectives with a minimizer and c0≥0, Theorem 3.2 gives a last-iterate function-gap rate.The proof combines nonincreasing distance to the minimizer with a convexity bound on the gradient norm.Source: §D.2 · p. 28
• Under the µ-PL condition, Theorem 3.2 gives a PL rate through a contraction factor in [0,1).The proof applies the PL condition to the descent inequality.Source: §D.2 · p. 28

E Experimental details

• Appendix E.2 specifies the deep-network training protocol, tuning grid, selection rule, and selected configurations.It also gives the CS-SAM pseudocode.Source: E Experimental details · p. 29

• The experiments use deterministic one-dimensional objectives with analytic values and gradients; the quartic uses ML(x)=4+|x|^3 and MH(x)=3+3x^4/4.The quadratic uses the same constant scale for both variants.Source: §E.1 · p. 29
• On the quadratic, from x0 = 3 with at most 500 updates, SAM (r = 1, η = 0.2) completes 500 updates, while CS-SAM (ρ = γ = 1/8) reaches the gradient stopping condition; its update count is unstated.The SAM two-cycle has amplitude 1/9.Source: §E.1 · p. 29
• On the quartic from x0=4, USAM’s trace ends after two completed updates as its next perturbed point exceeds the cutoff; both CS-SAM traces complete 10,000 updates.The USAM starting point exceeds the divergence threshold, while the cutoff is a numerical termination rule.Source: §E.1 · p. 30
• The distortion plots evaluate independent nonstationary points and compare each method’s relative gradient distortion with the CS-SAM bound e^(1/8)−1≈0.133.No outer update step enters this evaluation.Source: §E.1 · p. 30

Training protocol

• All methods train for 200 epochs with batch size 128 and compute two minibatch gradients per iteration, matching gradient-evaluation counts across methods.Experiments use ResNet-20 or ResNet-32 on CIFAR-10 or CIFAR-100, with cross-entropy loss, random-crop and horizontal-flip augmentation, and no momentum or weight decay.Source: §E.2 · p. 30

Baseline tuning

• SAM and USAM are tuned with constant and cosine-annealed outer-step schedules, while their radius or multiplier stays fixed during training.The cosine schedule anneals the outer step from η toward zero over 200 epochs without restarts, warm-up, or a positive floor.Source: §E.2 · p. 30

CS-SAM tuning

• CS-SAM tunes ML and MH curvature scales separately and uses no learning-rate schedule because both steps adapt through the scale.The ML scale is L0 + L1 ∥∇fi(x)∥ with L1 = 1; the MH scale is H0 + H1[fi(x) − ℓ⋆i]+ with H0 = 1 and ℓ⋆i = 0.Source: §E.2 · pp. 30–31

Seed reruns

• The selected configuration for each method and setting is rerun with seeds 3407 and 4096, and Table 3 reports mean and sample standard deviation of best test accuracy across three seeds.The configuration is first chosen by best test accuracy over 200 epochs on seed 3107.Source: §E.2 · p. 31

Selected configurations

• Table 4 records each method’s configuration selected on seed 3107 by best test accuracy over 200 epochs for every setting.The listed configurations are those selected in tuning stages (i) and (ii) and rerun in stage (iii).Source: Table 4 · p. 31, §E.2 · p. 31