Provable Edge-of-Stability for Adam on a One-Dimensional Quadratic

Yiman Fong; Heng Yang

Source-linked AI summary

Provable Edge-of-Stability for Adam on a One-Dimensional Quadratic

Yiman Fong, Heng Yang

arXiv:2608.20638v1Aug 21, 2026cs.LGcs.AImath.OC

TL;DR

Adam’s EoS mechanism is not fully understood, so this paper analyzes uncorrected Adam on a one-dimensional quadratic to isolate optimizer-induced dynamics from changing curvature. The analysis proves a restoring tendency toward the frozen stability boundary in broad regimes, while identifying periodic and persistently supercritical convergent trajectories that limit universal edge-seeking.

Problem
The dynamical mechanism underlying Adam’s widely observed edge-of-stability phenomenon remains insufficiently understood.
Method
The paper studies the exact discrete dynamics of uncorrected Adam on a one-dimensional quadratic with fixed curvature across parameter regimes.
Results
The analysis proves restoring behavior toward the frozen stability boundary in broad regimes and identifies strictly subcritical periodic orbits plus persistently supercritical convergent trajectories.
Takeaways & Limitations
Adam’s adaptive state can generate EoS intrinsically even with fixed loss curvature, while momentum can prevent universal convergence to the edge.
Takeaways & Limitations
The exact analysis is restricted to a one-dimensional quadratic, and exceptional trajectories can converge while remaining uniformly supercritical.

Abstract

from arXiv · show

The edge-of-stability (EoS) phenomenon of Adam has been widely observed, while its underlying dynamical mechanism is not yet fully understood. We study uncorrected Adam on a one-dimensional quadratic, a clean setting where constant curvature isolates the optimizer-induced dynamics behind the EoS. We characterize the resulting dynamics across the parameter space. In broad regimes, we prove that Adam exhibits a restoring tendency toward its frozen stability threshold $2(1+β_1)/[η(1-β_1)]$. We also identify settings in which this edge-seeking mechanism breaks down, including strictly subcritical periodic orbits and specially tuned trajectories that converge to the optimum while remaining uniformly supercritical. These results give a concrete dynamical explanation for Adam's EoS in a setting free of evolving loss geometry, while also exposing its limitations.

1 Introduction

The paper shows that uncorrected Adam can generate edge-of-stability behavior intrinsically on a one-dimensional quadratic, through a negative-feedback loop in its adaptive state. It also identifies parameter regimes and trajectories where this edge-seeking behavior fails.

The one-dimensional quadratic isolates optimizer-induced EoS dynamics while keeping the loss curvature fixed.
Adam’s sharpness oscillates around its frozen stability threshold in both neural-network training and the studied one-dimensional quadratic.
Above the edge, expansion raises v_t, shrinks the effective step size, and pushes normalized sharpness w_t downward; below it, contraction reverses these effects.
In broad regimes, an adaptive Lyapunov function certifies subcritical contraction up to an explicit cutoff W < 2, supporting restoration toward the edge.
The results establish an optimizer-induced mechanism for EoS with fixed curvature, while positive momentum introduces precise limitations and higher-dimensional extension remains future work.
Strictly subcritical periodic orbits can arise when ε = 0, while exceptional persistently misaligned trajectories can converge to the origin while remaining uniformly supercritical.

2 Mathematical formulation and the case of β1 = 0

Freezing Adam’s second moment reduces one-dimensional stability to the scalar normalized sharpness w, whose boundary is w = 2. In the β1 = 0 case, finite-passage results formalize restoring behavior toward this edge.

2.1 The stability threshold: The normalized sharpness satisfies 0 < w_t ≤ w_max, with w_max > 2 in essentially every realistic configuration because ε is tiny.The globally subcritical case w_max < 2 is treated separately.
2.1 The stability threshold: w = cη√v + ε determines frozen stability: the linear update is stable exactly when w < 2.The boundary is parameter-free after normalization.
2.1 The stability threshold: Frozen stability is only pointwise because v_t and w_t evolve along the adaptive trajectory.Thus, the frozen criterion does not alone determine adaptive-dynamics stability.
2.2 Illustrative example: β1 = 0 case: For β1 = 0, Adam reduces to RMSProp, the momentum variable disappears from the position recursion, and c = 1.This provides the simpler illustrative case.
2.2 Illustrative example: β1 = 0 case: For every w < 2 < w̄, trajectories cannot remain indefinitely below w or above w̄, formalizing restoration toward the edge.The levels can be chosen arbitrarily close to 2.
2.2 Illustrative example: β1 = 0 case: Below the edge, position contraction reduces second-moment forcing and raises w_t; above it, the reverse mechanism pushes w_t downward.This is the negative-feedback mechanism behind edge-seeking behavior.

3 Analysis of Adam with general parameters

The analysis characterizes positive-momentum Adam through normalized dynamics, proving restoring behavior toward the frozen edge in broad regimes while identifying periodic and convergent supercritical exceptions.

General parameter regimes: For the standard regime (β1, β2) = (0.9, 0.999), the frozen threshold remains w = 2, but momentum requires separate subcritical and supercritical analyses.The subcritical analysis controls joint position–momentum evolution, while the supercritical analysis uses their alignment.
General parameter regimes: The matrices A(w) are individually stable for 0 < w < 2, but their varying, noncommuting products require recursion-specific and sign-geometric arguments.The subcritical proof exploits restrictions on successive w_t values; the supercritical proof uses the signs of (x_t, h_t).
3.1 Subcritical analysis: For β1 = 0.9 and β2 = 0.999, the certified subcritical contraction cutoff is W ≈ 1.991, close to the frozen boundary.Within this region, an adaptive Lyapunov function decreases and trajectories cannot remain below any fixed w < W forever.
3.1.1 Existence of subcritical cycles: When ε = 0 under complementary parameter choices, Adam admits prime-period-four orbits satisfying w_t < 2 for every t.Thus, strictly subcritical cycling can prevent passage toward the edge.
3.2 Supercritical analysis: Aligned supercritical trajectories expand and increase v_t, forcing finite exit from every fixed band w ≥ 2 + δ and yielding lim inf_t→∞ w_t = 2.Misaligned trajectories can instead contract while remaining supercritical; specially chosen initial states converge to the origin with w_t approaching w_max.

4 Conclusion

The paper studies exact discrete uncorrected Adam dynamics on a one-dimensional quadratic to isolate optimizer-induced edge-of-stability behavior. It proves restoring behavior toward the frozen boundary in broad regimes, while showing that momentum permits strictly subcritical cycles and convergent uniformly supercritical trajectories.

The analysis identifies a restoring mechanism toward the frozen stability boundary.
With momentum, this behavior is established up to an explicit subcritical cutoff close to the threshold under standard parameter regimes.
Strictly subcritical periodic orbits and persistently supercritical yet convergent trajectories demonstrate limitations of universal edge convergence.
Extending the exact discrete analysis to higher-dimensional settings is identified as an important direction for future work.
The work places its mechanism alongside prior studies of edge-of-stability behavior, Adam convergence, and Adam’s discrete and continuous-time dynamics.

B The general Adam iteration and the frozen stability threshold

This section defines the general uncorrected Adam iteration and the preconditioned sharpness used to analyze frozen stability. The resulting Schur-stability condition yields the frozen threshold for every positive-curvature mode.

B.1 General Adam iteration: Uncorrected Adam maintains coordinatewise momentum and second-moment estimates for a twice continuously differentiable objective, with β1, β2, η, ε, and an initial state as parameters.The one-dimensional quadratic is a special case of this general iteration.
B.2 Frozen stability threshold: Freezing the second-moment variable produces an operator on position and momentum whose preconditioned Hessian determines the stability boundary.
B.2 Frozen stability threshold: The frozen linearization is Schur stable exactly when every preconditioned-Hessian eigenvalue satisfies 0 < η(1 − β1)λ < 2(1 + β1).
B.2 Frozen stability threshold: Equivalently, the largest preconditioned-Hessian eigenvalue must satisfy λmax(eH_t) < 2(1 + β1)/(η(1 − β1)).

D Proofs for the β1 = 0 case

For β1 = 0, the proofs establish finite movement toward the edge from below and finite exit from any uniformly supercritical band above it.

Summary: The β1 = 0 results provide finite-time bounds for approaching the threshold from either side.The appendix explicitly records finite passage toward a strict subcritical target and finite supercritical exit.
Subcritical stage: The subcritical-stage argument shows that trajectories below the target eventually reach the strict subcritical threshold.The proof controls the position and second moment over stopped trajectories, then uses the second-moment recursion to force w_T+n ≥ b̄w.
Supercritical stage: The supercritical-stage argument shows that a nonzero trajectory cannot remain indefinitely above 2 + δ.While the band persists, |x_T+j| grows at least geometrically, forcing the second moment upward until the band condition becomes impossible.
Supercritical stage: A trajectory that survives in the supercritical band through time T + n must satisfy an increasingly restrictive second-moment bound.The survival implication is then combined with the divergence of the lower bound generated by geometric position growth.

E.1 Proof of Proposition 3.1

The proof of Proposition 3.1 constructs an adaptive Lyapunov function that strictly contracts below an explicit cutoff W < 2.

Lyapunov construction: The Lyapunov construction uses a one-step quadratic inequality for the state transition matrix.The matrix D(y, Y; w_t+1) is formed from a weighted quadratic form and its propagated version.
Lyapunov construction: For 0 < w_t+1 < 2, Sylvester’s criterion establishes positivity of the relevant quadratic form.The proof then verifies positivity of the factors needed for the contraction estimate.
Contraction result: The resulting cutoff satisfies 0 < W < 2, and the Lyapunov quantity strictly decreases for every nonzero state.This establishes contraction throughout the certified subcritical region.
Contraction result: The cutoff also admits an expansion for fixed β1 and wmax.The appendix records this expansion as a refinement of Proposition 3.1.

E.2 Proof of Corollary 3.2

The proof of Corollary 3.2 shows that trajectories below the Lyapunov cutoff cannot remain there forever when wmax > 2.

Proof by contradiction: Assuming the normalized sharpness remains bounded above by a subcritical value leads to a contradiction.The second-moment recursion drives w_T+n toward wmax > 2, exceeding the assumed bound.
Quantitative passage: The quantitative corollary gives an explicit finite-time passage toward the subcritical cutoff under wmax > 4.For an initial gap δ below W, the proof bounds the number of steps needed to cross the target.
Exceptional periodic orbit: The construction of strictly subcritical four-cycles shows that the edge-seeking conclusion is not universal when ε = 0.The orbit has four distinct position values and remains strictly below w = 2 at every step.

F.1 Proof of Lemma 3.4

The proof of Lemma 3.4 separates aligned supercritical trajectories from misaligned ones, yielding expansion in the first case and a route to convergence in the second.

Aligned trajectories: Aligned supercritical steps reverse signs and expand the position magnitude by a factor exceeding w_t+1 − 1.This gives a direct lower bound on position growth while alignment persists.
Aligned trajectories: An aligned trajectory cannot remain uniformly supercritical indefinitely, so its lower limit is 2 if it stays supercritical.The finite-exit argument rules out lim inf_t w_t > 2.
Misaligned trajectories: A specially tuned misaligned trajectory can remain misaligned at every finite time through a nested shooting construction.The construction selects an initial momentum parameter from nested survival intervals.
Misaligned trajectories: The persistent misaligned trajectory converges geometrically to (0, 0, 0) while w_t approaches wmax.Thus convergence can occur while the trajectory remains uniformly supercritical.
Numerical illustration: The numerical illustration shows w_t increasing toward wmax = 4 while |x_12| < 10^-9 without leaving the supercritical region.The orbit remains misaligned over the displayed steps and follows the certified decay envelope.

G.1 Global convergence in the globally subcritical zero-momentum regime

In the globally subcritical regime, Adam converges globally and exponentially to the origin under the stated conditions, including with positive momentum. For zero momentum, this follows from a scalar contraction; positive momentum requires a quadratic Lyapunov argument.

Zero momentum: For β1 = 0 and wmax < 2, every trajectory converges exponentially to (0, 0, 0).The position contracts geometrically, while the second moment decays through its recursion.
Proof mechanism: For β1 = 0, the scalar position recursion admits a common contraction factor once vt is bounded.This implication does not extend directly to positive momentum, whose adaptive dynamics are two-dimensional.
Positive momentum: When 0 < β1 < 1 and wmax < 2, a quadratic Lyapunov function yields global exponential convergence to the origin.The proof establishes a strict contraction on an invariant compact set.
Positive momentum: In the strictly subcritical regime, wt converges to wmax while (xt, mt, vt) converges to (0, 0, 0).Thus the adaptive normalized sharpness approaches the limiting subcritical boundary from below.
Proof mechanism: For positive momentum, every transition satisfies 0 < wt+1 ≤ Φ(wt), and the Lyapunov construction controls the resulting coupled dynamics.The proof uses the transition bound to define an invariant compact region and then applies a strict quadratic inequality.

H Experimental details for Figure 1

Figure 1 compares Adam’s edge-of-stability behavior in a neural-network regression experiment with uncorrected Adam on a one-dimensional quadratic. The quadratic experiment isolates the normalized sharpness oscillation around the parameter-free boundary w = 2.

Figure 1 setup: Standard bias-corrected Adam trains a 16-hidden-unit tanh network on y = sin(2x) over 32 inputs in [−2, 2].The experiment uses β1 = 0.9, β2 = 0.999, η = 10−2, and ε = 10−8; Hessians are estimated every 40 steps by central finite differences.
Figure 1 result: The neural-network experiment’s preconditioned sharpness equilibrates near S⋆ = 2(1 + β1)/[η(1 − β1)] = 38/η.The plotted sharpness uses the bias-corrected second-moment estimate.
Figure 1 setup: Uncorrected Adam on the quadratic uses η = 1, wmax = 10, x0 = 0.1, m0 = 0, and v0 = 0.01.The corresponding stabilizer is ε = cη/10, with the same β1 and β2 values as the neural-network experiment.
Figure 1 result: In the quadratic experiment, the normalized sharpness wt oscillates around the parameter-free boundary w = 2.The accompanying mechanism is that supercritical steps inflate vt and push wt downward, whereas subcritical steps deflate vt and push wt upward.