PaperlayerView exact revision on arXiv

Source-linked AI summary

The Multiple Timescales of Gradient Descent on the Edge of Stability: A Perturbative Derivation of the Central Flow

Raphaël Berthier

arXiv:2609.01034v1cs.LGmath.OCstat.ML

TL;DR

The paper addresses the heuristic derivation of the central flow as a model of gradient descent at the edge of stability. It introduces a perturbative, multiple-scales framework for deriving the flow and self-stabilization dynamics, finding three separated timescales and formal central-flow and fluctuation terms. The analysis remains formal and has scope boundaries tied to stable-valley behavior and energy levels.

  • Problem

    The central flow is empirically accurate, but its existing derivation is heuristic, motivating a regime in which it can arise as a limit of gradient descent.

  • Method

    The paper assumes f = g + εh and uses formal multiple-scale analysis to treat gradient descent as a singularly perturbed system with fast, intermediate, and slow timescales.

  • Results

    The central flow appears as the leading-order slow-timescale term, while oscillations and self-stabilization fluctuations appear at next order; the central flow is more coarse-grained than competing approximations.

  • Takeaways & Limitations

    The framework relates central-flow and self-stabilization behavior to sharp-valley structure and provides a formal basis for analyzing gradient descent at the edge of stability.

  • Takeaways & Limitations

    The derivations are formal, and the analysis assumes stable-valley behavior while treating the self-stabilization system at energy levels of order 1.

Abstract

from arXiv · show

The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as $f = g + \varepsilon h$; in the limit $\varepsilon \to 0$, the dynamics of gradient descent with learning rate $η$ converge to the gradient flow of $h$ constrained to the minimizers of $g$ of sharpness at most $2/η$. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in $\varepsilon$. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in $\varepsilon$: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

1 Introduction

The paper addresses the heuristic derivation of the central flow by introducing a perturbative regime where it becomes the formal limit of gradient descent. Multiple scales separate sharp-direction oscillations, self-stabilization, and slow valley dynamics, while extending analysis to fluctuation behavior and competing approximations.

  • Gradient descent iterates form a rough path because sharp-direction oscillations and self-stabilization fluctuations prevent direct approximation by a smooth loss gradient flow.
  • The central flow averages these fluctuations and is the gradient flow of the loss constrained to the stable region where sharpness is below 2/η.
  • The sharp valley assumption decomposes the loss as f = g + εh, with g defining a valley and h inducing dynamics along it.
  • As ε → 0, gradient descent becomes a singularly perturbed system with fast t1 oscillations, intermediate t2 self-stabilization, and slow t3 central-flow dynamics.
  • The method of multiple scales makes the central flow the leading-order term and places oscillations and self-stabilization fluctuations in the next-order term.The derivation is formal: it matches asymptotic orders without proving convergence or bounding the remainder.
  • With multiple edge eigenvalues, the paper derives the self-stabilization system and explains persistent fluctuations; with one, it computes fluctuation-energy variations.
  • Compared with competing continuous-time approximations, the central flow is more coarse-grained because it depends only on t3 rather than t2 and t3.

2 Setting

The setting models a loss with a valley of minimizers and sharp directions orthogonal to that valley. For tractability, the valley is assumed to be linear, with positive curvature in the sharp directions and a stable subset defined by the 2/η threshold.

  • The loss decomposes as f(z) = g(z) + εh(z), where g defines the valley V of minimizers and ε is small.
  • The analysis assumes a linear valley through an orthogonal decomposition Z = X ⊕⊥ Y, with z = (x, y).
  • In a manifold setting, the same decomposition would serve as local coordinates for the minimizer manifold V.
  • The Hessian of g in directions orthogonal to the valley is assumed positive definite along V.
  • The stable valley consists of points whose sharpness is below 2/η.

3 Convergence to the central flow

The paper formally derives convergence of gradient descent to the central flow under a sharp-valley perturbative regime, using slow-time rescaling and multiple-scales analysis. The result applies to codimension-one and arbitrary-codimension valleys under stability assumptions, but remains formal rather than rigorous.

  • Setup: The rescaled iterates use the slow time t3 = εk, which captures dynamics along the valley as ε → 0.The continuous-time parameter is implemented by k = ⌊t3/ε⌋.
  • Codimension one: For codimension-one valleys, the limiting trajectory follows the gradient flow of h constrained to the stable valley VS.The result assumes convergence of the rescaled iterates and ηS(0, y0) ≤ 2.
  • Codimension one: The codimension-one constraint is represented by a nonnegative Lagrange multiplier satisfying complementarity slackness.The multiplier vanishes away from the boundary and may be positive when ηS(0, y0) = 2.
  • Arbitrary codimension: For arbitrary codimension, the limiting trajectory follows the gradient flow of h constrained by ηS(0, y0) ≼ 2 IdX.The matrix-valued multiplier Σ is positive semidefinite and encodes the active stability constraint.
  • Higher-order terms: The method of multiple scales also yields next-order oscillation and self-stabilization terms beyond the leading central-flow limit.These corrections are developed for codimension-one valleys and the general case.

4 Minimal example

The minimal example shows gradient descent progressing along a valley until reaching the stability boundary, where oscillations and self-stabilization fluctuations emerge. Multiple-scales analysis identifies the central flow at leading order and the self-stabilization dynamics at the next order.

  • 4.1 Introduction and experiments: The valley dynamics move toward increasing y until reaching the edge of stability at y = 2/η.The sharpness increases progressively along the valley and reaches the threshold y = 2/η.
  • 4.1 Introduction and experiments: As ε decreases, the iterates converge to the central flow: gradient flow of h constrained to the stable valley VS = {(0, y) : y ≤ 2/η}.The limiting trajectory follows the valley until it reaches the stability constraint and then remains on its boundary.
  • 4.2 Method of multiple scales: Multiple scales t1 = k, t2 = ε1/2k, and t3 = εk separate fast oscillations, intermediate self-stabilization, and slow valley dynamics.The method treats these scales as independent before substituting their definitions into the expansion.
  • 4.2 Method of multiple scales: At the stability edge, x1 oscillates in t1 while its amplitude and y1 evolve through the self-stabilization system on t2.The oscillation has the form x1 = (−1)t1a1(t2, t3), with a1 independent of t1.
  • 4.3 Energy variations in self-stabilization: The self-stabilization dynamics preserve fluctuation energy over t2, so the energy evolves only on the slow timescale t3.For a single edge eigenvalue, the paper derives the slow energy variation and notes that the analysis is intended for energy levels of order 1.
  • 4.3 Energy variations in self-stabilization: Removing the −x term from h can let the trajectory leave the stable valley, after which delayed self-stabilization may cause divergence.The paper characterizes this as a failure of the central-flow prediction in the minimal example.

5 Valleys of codimension one

For valleys of codimension one, the analysis extends the central-flow and self-stabilization description beyond the minimal example. Averaging over the intermediate timescale gives a precise interpretation of the fluctuation average used in the central flow.

  • 5 Valleys of codimension one: For a one-dimensional transverse space, the leading valley dynamics remain on the slow timescale t3, while the first correction depends on t2 and t3.The result assumes the leading trajectory stays in the stable valley.
  • 5 Valleys of codimension one: At the edge of stability, the transverse correction oscillates as x1 = (−1)t1a1(t2, t3), and its projected amplitude participates in the self-stabilization system.The scalar transverse case recovers the corresponding minimal-example structure up to rescaling.
  • 5 Valleys of codimension one: Neural-network experiments show decreasing fluctuation amplitude with one edge eigenvalue and larger stabilized fluctuations when a second eigenvalue reaches the edge.The experiment uses a two-hidden-layer width-256 GELU multilayer perceptron on a CIFAR-10 subset.
  • 5 Valleys of codimension one: The fluctuation average σ2 is an average over t2 at fixed t3, making the central flow’s local time-averaging operator precise.This identifies the averaged squared amplitude with the quantity used in the central-flow derivation.

6 Valleys of arbitrary codimension

For valleys with arbitrary codimension, the self-stabilization system acts on the critical subspace associated with all edge eigenvalues. Generically, multiple edge eigenvalues prevent convergence to a fixed point and produce persistent fluctuations.

  • 6 Valleys of arbitrary codimension: The critical subspace C is the kernel of 2 IdX − ηS(0, y0), and the critical component satisfies PCx1 = (−1)t1a1.The orthogonal complement of the first correction vanishes in the stated limit.
  • 6 Valleys of arbitrary codimension: Averaging a1a1⊤ over t2 at fixed t3 defines the covariance-like quantity Σ used in the generalized central-flow description.This makes the local time average precise through the separation of intermediate and slow timescales.
  • 6 Valleys of arbitrary codimension: With one edge eigenvalue, the generalized system reduces to the codimension-one system and exhibits periodic behavior in t2.In this case, the critical subspace is one-dimensional and the relevant quantities are scalars.
  • 6 Valleys of arbitrary codimension: With multiple edge eigenvalues, a fixed point is generically absent, so the fluctuation matrix must evolve persistently to maintain the required average.This produces richer self-stabilization dynamics than in the one-eigenvalue case.
  • 6 Valleys of arbitrary codimension: When a second eigenvalue reaches the edge, fluctuations increase and persist, matching the reported neural-network phenomenology.The paper contrasts this with energy loss and convergence toward a fixed point when only one eigenvalue is at the edge.
  • 6 Valleys of arbitrary codimension: The loss contains an order-ε valley contribution plus an edge-of-stability increase proportional to ∥a1∥2/η, producing t2-scale spikes.Averaging the spikes over t2 yields the corresponding slow loss description.

7 Related work

Prior work supports the sharp-valley picture and develops multiple approximations of edge-of-stability dynamics, while this paper uses multiple timescales to clarify their relationships and tradeoffs.

  • Sharp valley structure: The sharp-valley picture is consistent with empirical observations including anti-alignment, low-rank Hessians, and plots around consecutive iterates.These observations motivate treating valley structure as a working hypothesis.
  • Edge of stability: Earlier edge-of-stability analyses explain the phenomenon differently; this work instead assumes that leading-order dynamics remain in the stable region.
  • Valley dynamics: Related valley analyses describe Riemannian sharpness descent and orthogonal bifurcations, with their subcritical regime corresponding to h = 0 here.
  • Alternative approximations: Several post-central-flow approximations capture slow dynamics and attempt to model self-stabilization, but their derivations average at different timescales.
  • Alternative approximations: Averaging over t1 and t2 yields the simplest central flow, whereas averaging over t1 only retains more self-stabilization detail at greater conceptual complexity.

8 Conclusion

The paper connects the central flow and self-stabilization to sharp-valley structure and develops a multiple-scales framework for edge-of-stability dynamics. It remains formal, and explaining why such valleys arise in deep learning is still open.

  • Conclusion: The paper’s conceptual contribution is to relate the central flow and self-stabilization mechanism to a sharp-valley loss landscape.
  • Conclusion: Figure 10 compares the central flow, rod flow, edge gradient descent, and free energy flow in the setting of Fig. 3.The rod flow may fail to detect the start of the edge-of-stability phase.
  • Conclusion: The origin of sharp-valley structure during deep-learning training remains largely an open problem.
  • Conclusion: Analytically, the paper provides a multiple-scales framework for studying gradient descent at the edge of stability, while noting that rigorous singular-perturbation formalization and curved valleys remain future directions.
  • Conclusion: The framework’s systematic structure is intended for future analysis of inertial and adaptive algorithms and may inform the design of new methods.

A Derivation of Result 4.4

The derivation expands the discrete dynamics across the fast oscillation scale and removes secular terms order by order. This yields conditions governing higher-order corrections and the fluctuation energy.

  • Order ε3/2 expansion: The expansion computes ε3/2-order terms in the discrete update and identifies their dependence on the fast variable t1.
  • Edge-of-stability ansatz: At the edge of stability, the leading sharp-direction fluctuation has the alternating form x1 = (−1)^t1a1(t2, t3).
  • Secular-term removal: Avoiding secular terms in the t1-dependent equations imposes solvability conditions on the correction amplitudes, including a2.
  • Intermediate-scale solvability: The t2-dependent right-hand side is periodic through a1 and y1, so boundedness requires its average to vanish.The derivation uses the lemma that a periodic function has a bounded primitive exactly when its average is zero.
  • Energy and phase: Ensuring F = y1y2 + a2 is non-secular suffices to compute the energy variation, but an additional condition is needed to determine the phase shift of self-stabilization fluctuations.That phase shift is observed in the bottom plots of Fig. 5.

B Derivation of Result 5.1

The derivation separates stable-region decay from edge-of-stability oscillations and imposes solvability conditions across the multiple scales. It obtains constrained valley dynamics and a periodic self-stabilization system.

  • Setup: The derivation assumes a valley of minimizers where derivatives of g in the orthogonal direction vanish, simplifying the expanded dynamics.
  • Multiple-scale decomposition: At order one, the slow variable satisfies y0 = y0(t2, t3), and the analysis then distinguishes the stable region from the edge of stability.
  • Stable region: When ηS < 2, the fast fluctuation decays as x1 → 0 for t1 → ∞.
  • Edge of stability: When ηS = 2, the fast fluctuation persists as x1 = (−1)^t1a1(t2, t3), producing edge-of-stability oscillations.
  • Self-stabilization: The derivation distinguishes edge and interior behavior through the sharpness condition ηS = 2 versus ηS < 2.
  • Self-stabilization: At the edge, solvability conditions determine the intermediate-scale evolution, while the resulting self-stabilization system makes a1 periodic in t2.
  • Stable-region dynamics: In the relative interior of the stable region, the dynamics follow the gradient flow of h constrained to the valley.

C Derivation of Result 6.1

The derivation decomposes the dynamics into critical and noncritical eigendirections, removes secular terms, and obtains the self-stabilization system through averaging on the intermediate timescale.

  • The spectrum splits into the critical subspace C, associated with eigenvalue 2 of ηS, and its complement C⊥, containing eigenvalues in (0, 2).
  • Noncritical fluctuations x⊥_1 vanish as the fast timescale t1 tends to infinity, leaving the critical component as the persistent oscillatory part.
  • The fast dynamics are analyzed through difference equations whose bounded solutions require cancellation of the forcing term that would otherwise create secular growth.
  • The resulting next-order dynamics couple an oscillatory amplitude a1 with y1 on the intermediate timescale t2, yielding the self-stabilization equations.
  • For multiple edge eigenvalues, the derivation defines the averaged covariance through long-time averages or accumulation points of bounded amplitudes.

D Derivations of Corollary 6.2

The derivation expands the loss near a minimizer of g using scaled x and y variables. The resulting first-order expression combines the perturbation h with a quadratic contribution, while sharp-direction components decay toward the edge eigenspace.

  • The expansion substitutes ε-scaled x and y perturbations into g and evaluates εh near the minimizer.
  • At (0, y0), both gradients of g vanish and g reaches its minimum.
  • The loss expands as f(x, y) = min g + εf2 + o(ε), with f2 beginning with h(0, y0) and a quadratic term.
  • The component orthogonal to the edge eigenspace decays as t1 grows, leaving the eigenspace associated with eigenvalue 2/η.
Loading 2609.01034v1…