Provable Edge-of-Stability for Adam on a One-Dimensional Quadratic

• The one-dimensional quadratic isolates optimizer-induced EoS dynamics while keeping the loss curvature fixed.Source: §1 · pp. 1–2
• Adam’s sharpness oscillates around its frozen stability threshold in both neural-network training and the studied one-dimensional quadratic.Source: §1 · p. 1, Figure 1 · p. 2
• Above the edge, expansion raises v_t, shrinks the effective step size, and pushes normalized sharpness w_t downward; below it, contraction reverses these effects.Source: §1 · pp. 1–2, Figure 1 · p. 2
• In broad regimes, an adaptive Lyapunov function certifies subcritical contraction up to an explicit cutoff W < 2, supporting restoration toward the edge.Source: §1 · p. 2
• The results establish an optimizer-induced mechanism for EoS with fixed curvature, while positive momentum introduces precise limitations and higher-dimensional extension remains future work.Source: §1 · p. 2
• Strictly subcritical periodic orbits can arise when ε = 0, while exceptional persistently misaligned trajectories can converge to the origin while remaining uniformly supercritical.Source: §1 · p. 2

2.1 The stability threshold

• The normalized sharpness satisfies 0 < w_t ≤ w_max, with w_max > 2 in essentially every realistic configuration because ε is tiny.The globally subcritical case w_max < 2 is treated separately.Source: §2.1 · p. 3
• w = cη√v + ε determines frozen stability: the linear update is stable exactly when w < 2.The boundary is parameter-free after normalization.Source: §2.1 · p. 3
• Frozen stability is only pointwise because v_t and w_t evolve along the adaptive trajectory.Thus, the frozen criterion does not alone determine adaptive-dynamics stability.Source: §2.1 · p. 3

2.2 Illustrative example: β1 = 0 case

• For β1 = 0, Adam reduces to RMSProp, the momentum variable disappears from the position recursion, and c = 1.This provides the simpler illustrative case.Source: §2.2 · p. 3
• For every w < 2 < w̄, trajectories cannot remain indefinitely below w or above w̄, formalizing restoration toward the edge.The levels can be chosen arbitrarily close to 2.Source: §2.2 · p. 4
• Below the edge, position contraction reduces second-moment forcing and raises w_t; above it, the reverse mechanism pushes w_t downward.This is the negative-feedback mechanism behind edge-seeking behavior.Source: §2.2 · p. 4

General parameter regimes

• For the standard regime (β1, β2) = (0.9, 0.999), the frozen threshold remains w = 2, but momentum requires separate subcritical and supercritical analyses.The subcritical analysis controls joint position–momentum evolution, while the supercritical analysis uses their alignment.Source: §3 · p. 4
• The matrices A(w) are individually stable for 0 < w < 2, but their varying, noncommuting products require recursion-specific and sign-geometric arguments.The subcritical proof exploits restrictions on successive w_t values; the supercritical proof uses the signs of (x_t, h_t).Source: §3 · p. 4

3.1 Subcritical analysis

• For β1 = 0.9 and β2 = 0.999, the certified subcritical contraction cutoff is W ≈ 1.991, close to the frozen boundary.Within this region, an adaptive Lyapunov function decreases and trajectories cannot remain below any fixed w < W forever.Source: §3.1 · pp. 4–5

3.1.1 Existence of subcritical cycles

• When ε = 0 under complementary parameter choices, Adam admits prime-period-four orbits satisfying w_t < 2 for every t.Thus, strictly subcritical cycling can prevent passage toward the edge.Source: §3.1.1 · p. 5

3.2 Supercritical analysis

• Aligned supercritical trajectories expand and increase v_t, forcing finite exit from every fixed band w ≥ 2 + δ and yielding lim inf_t→∞ w_t = 2.Misaligned trajectories can instead contract while remaining supercritical; specially chosen initial states converge to the origin with w_t approaching w_max.Source: §3.2 · pp. 5–6

• The analysis identifies a restoring mechanism toward the frozen stability boundary.Source: §4 · p. 6
• With momentum, this behavior is established up to an explicit subcritical cutoff close to the threshold under standard parameter regimes.Source: §4 · p. 6
• Strictly subcritical periodic orbits and persistently supercritical yet convergent trajectories demonstrate limitations of universal edge convergence.Source: §4 · p. 6
• Extending the exact discrete analysis to higher-dimensional settings is identified as an important direction for future work.Source: §4 · p. 6
• The work places its mechanism alongside prior studies of edge-of-stability behavior, Adam convergence, and Adam’s discrete and continuous-time dynamics.Source: A Related Work · pp. 8–9

B.1 General Adam iteration

• Uncorrected Adam maintains coordinatewise momentum and second-moment estimates for a twice continuously differentiable objective, with β1, β2, η, ε, and an initial state as parameters.The one-dimensional quadratic is a special case of this general iteration.Source: B The general Adam iteration and the… · p. 9

B.2 Frozen stability threshold

• Freezing the second-moment variable produces an operator on position and momentum whose preconditioned Hessian determines the stability boundary.Source: B The general Adam iteration and the… · p. 9
• The frozen linearization is Schur stable exactly when every preconditioned-Hessian eigenvalue satisfies 0 < η(1 − β1)λ < 2(1 + β1).Source: B The general Adam iteration and the… · p. 9
• Equivalently, the largest preconditioned-Hessian eigenvalue must satisfy λmax(eH_t) < 2(1 + β1)/(η(1 − β1)).Source: B The general Adam iteration and the… · p. 9

Summary

• The β1 = 0 results provide finite-time bounds for approaching the threshold from either side.The appendix explicitly records finite passage toward a strict subcritical target and finite supercritical exit.Source: D Proofs for the β1 = 0 case · pp. 10, 14

Subcritical stage

• The subcritical-stage argument shows that trajectories below the target eventually reach the strict subcritical threshold.The proof controls the position and second moment over stopped trajectories, then uses the second-moment recursion to force w_T+n ≥ b̄w.Source: D Proofs for the β1 = 0 case · pp. 10, 12

Supercritical stage

• The supercritical-stage argument shows that a nonzero trajectory cannot remain indefinitely above 2 + δ.While the band persists, |x_T+j| grows at least geometrically, forcing the second moment upward until the band condition becomes impossible.Source: D Proofs for the β1 = 0 case · pp. 13–15
• A trajectory that survives in the supercritical band through time T + n must satisfy an increasingly restrictive second-moment bound.The survival implication is then combined with the divergence of the lower bound generated by geometric position growth.Source: D Proofs for the β1 = 0 case · p. 15

Lyapunov construction

• The Lyapunov construction uses a one-step quadratic inequality for the state transition matrix.The matrix D(y, Y; w_t+1) is formed from a weighted quadratic form and its propagated version.Source: §E.1 · pp. 16–17
• For 0 < w_t+1 < 2, Sylvester’s criterion establishes positivity of the relevant quadratic form.The proof then verifies positivity of the factors needed for the contraction estimate.Source: §E.1 · pp. 17–19

Contraction result

• The resulting cutoff satisfies 0 < W < 2, and the Lyapunov quantity strictly decreases for every nonzero state.This establishes contraction throughout the certified subcritical region.Source: §E.1 · pp. 17–18
• The cutoff also admits an expansion for fixed β1 and wmax.The appendix records this expansion as a refinement of Proposition 3.1.Source: §E.1 · p. 19

Proof by contradiction

• Assuming the normalized sharpness remains bounded above by a subcritical value leads to a contradiction.The second-moment recursion drives w_T+n toward wmax > 2, exceeding the assumed bound.Source: §E.2 · pp. 19–20

Quantitative passage

• The quantitative corollary gives an explicit finite-time passage toward the subcritical cutoff under wmax > 4.For an initial gap δ below W, the proof bounds the number of steps needed to cross the target.Source: §E.2 · p. 20

Exceptional periodic orbit

• The construction of strictly subcritical four-cycles shows that the edge-seeking conclusion is not universal when ε = 0.The orbit has four distinct position values and remains strictly below w = 2 at every step.Source: §E.3 · p. 21

Aligned trajectories

• Aligned supercritical steps reverse signs and expand the position magnitude by a factor exceeding w_t+1 − 1.This gives a direct lower bound on position growth while alignment persists.Source: §F.1 · p. 21
• An aligned trajectory cannot remain uniformly supercritical indefinitely, so its lower limit is 2 if it stays supercritical.The finite-exit argument rules out lim inf_t w_t > 2.Source: §F.2 · p. 22

Misaligned trajectories

• A specially tuned misaligned trajectory can remain misaligned at every finite time through a nested shooting construction.The construction selects an initial momentum parameter from nested survival intervals.Source: §F.3 · pp. 23–24
• The persistent misaligned trajectory converges geometrically to (0, 0, 0) while w_t approaches wmax.Thus convergence can occur while the trajectory remains uniformly supercritical.Source: §F.3 · p. 24, §F.4 · p. 24

Numerical illustration

• The numerical illustration shows w_t increasing toward wmax = 4 while |x_12| < 10^-9 without leaving the supercritical region.The orbit remains misaligned over the displayed steps and follows the certified decay envelope.Source: §F.4 · pp. 24–25

Zero momentum

• For β1 = 0 and wmax < 2, every trajectory converges exponentially to (0, 0, 0).The position contracts geometrically, while the second moment decays through its recursion.Source: §G.1 · pp. 25–26

Proof mechanism

• For β1 = 0, the scalar position recursion admits a common contraction factor once vt is bounded.This implication does not extend directly to positive momentum, whose adaptive dynamics are two-dimensional.Source: §G.1 · p. 26

Positive momentum

• When 0 < β1 < 1 and wmax < 2, a quadratic Lyapunov function yields global exponential convergence to the origin.The proof establishes a strict contraction on an invariant compact set.Source: §G.2 · pp. 26–27
• In the strictly subcritical regime, wt converges to wmax while (xt, mt, vt) converges to (0, 0, 0).Thus the adaptive normalized sharpness approaches the limiting subcritical boundary from below.Source: §G.2 · p. 26

Proof mechanism

• For positive momentum, every transition satisfies 0 < wt+1 ≤ Φ(wt), and the Lyapunov construction controls the resulting coupled dynamics.The proof uses the transition bound to define an invariant compact region and then applies a strict quadratic inequality.Source: §G.2 · p. 27

Figure 1 setup

• Standard bias-corrected Adam trains a 16-hidden-unit tanh network on y = sin(2x) over 32 inputs in [−2, 2].The experiment uses β1 = 0.9, β2 = 0.999, η = 10−2, and ε = 10−8; Hessians are estimated every 40 steps by central finite differences.Source: H Experimental details for Figure 1 · p. 28

Figure 1 result

• The neural-network experiment’s preconditioned sharpness equilibrates near S⋆ = 2(1 + β1)/[η(1 − β1)] = 38/η.The plotted sharpness uses the bias-corrected second-moment estimate.Source: H Experimental details for Figure 1 · p. 28

Figure 1 setup

• Uncorrected Adam on the quadratic uses η = 1, wmax = 10, x0 = 0.1, m0 = 0, and v0 = 0.01.The corresponding stabilizer is ε = cη/10, with the same β1 and β2 values as the neural-network experiment.Source: H Experimental details for Figure 1 · p. 28

Figure 1 result

• In the quadratic experiment, the normalized sharpness wt oscillates around the parameter-free boundary w = 2.The accompanying mechanism is that supercritical steps inflate vt and push wt downward, whereas subcritical steps deflate vt and push wt upward.Source: H Experimental details for Figure 1 · p. 28