Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

Problem setting

• Power-law kernel regression makes momentum analytically tractable because its infinite-dimensional spectrum spans many learning timescales, with β controlling feature-spectrum decay and s controlling target-energy distribution.The setting extends the functional scaling law framework to momentum methods.Source: §1 · p. 2

Analysis framework

• The analysis optimizes final-step excess risk over admissible learning rates and momentum factors under the one-pass constraint T = D/B.The hierarchy first identifies the stable hyperparameter region, then derives risk scaling laws, and finally optimizes final-step risk subject to admissibility and the fixed data budget.Source: §1 · p. 2

Risk stability

• Momentum stability requires larger batches as momentum strengthens, with the compensation scaling as B for Polyak and B^β for Nesterov, so Nesterov permits substantially higher momentum in large-batch training.The stability contribution characterizes the critical learning rate as the largest stable learning rate.Source: §1 · pp. 2–3

Scaling laws for risk dynamics

• Within the stable region, risk evolves from an early exponential transient through power-law decay to a noise floor, with Nesterov reducing noise accumulation through curvature-adaptive damping from its look-ahead mechanism.Polyak preserves the SGD signal–noise tradeoff at the effective learning rate ηρ := η/(1 −ρ), whereas Nesterov suppresses the noise floor at large ηρ.Source: §1 · p. 3

Data scaling and batch-size phase diagram

• Polyak preserves the best small-batch data-scaling exponent over a wider batch-size range, while Nesterov outperforms Polyak at intermediate large batches and both momentum methods outperform SGD at ultra-large batches.The phase diagram has three regimes separated by b1 < b2 < b3; Nesterov achieves its best data-scaling rate at b = b2.Source: §1 · p. 3

Why momentum helps large-batch training

• Polyak enlarges the admissible per-sample effective learning rate from SGD’s 1/B scale to order one, extending the signal–noise tradeoff without changing its optimum, whereas Nesterov improves the tradeoff itself.At ultra-large batch sizes, the limited number of sequential updates becomes the dominant bottleneck and the two momentum methods recover the same signal-limited scaling.Source: §1 · pp. 3–4

Theory of momentum

• The study compares Polyak and Nesterov across batch-size regimes under a fixed data budget, addressing a gap in existing momentum theory.Prior analyses emphasize accelerated convergence, Lyapunov functions, continuous-time limits, stochastic convergence, stability, and asymptotic behavior, but do not characterize batch-size-dependent momentum roles.Source: §2 · p. 4

Scaling-law theory

• The work extends scaling-law theory to constant-order label noise and variable batch sizes, contrasting canonical Polyak and Nesterov with related fixed-batch noiseless analyses.Ferbach et al. (2026a) study noiseless power-law random features with fixed batch size, finding that Polyak preserves SGD exponents while adapted Nesterov-type methods can improve them.Source: §2 · p. 4

Stability of gradient-based methods

• The work derives sharp risk-stability boundaries whose joint dependence on batch size and momentum reveals stronger batch-size compensation for high-momentum Nesterov.Existing stability studies connect admissible hyperparameters to implicit bias, generalization, and optimization geometry, while momentum’s batch-size dependence remains less understood.Source: §2 · p. 4

Concurrent work

• Unlike concurrent lower-bound results, the work derives sharp full-risk scaling laws and an optimized batch-size phase diagram where Nesterov outperforms Polyak at large batch sizes.The phase diagram makes the data-scaling exponent explicit as a function of batch size, whereas matching upper bounds remain open in Morwani et al. (2026).Source: §2 · p. 4

3 Preliminaries

• The framework uses equivalence-up-to-constants notation and standard inner-product and 2-norm notation throughout.Source: §3 · p. 5

3.1 Power-law kernel regression

• Power-law kernel regression uses Gaussian features in a Hilbert space, a linear student model, and excess risk measured through the covariance operator.Because the algorithms are orthogonally equivariant, analysis can be performed in the covariance eigenbasis.Source: §3.1 · p. 5
• The capacity exponent β controls spectral decay, while source exponent s measures task difficulty; smaller s concentrates target energy in low-curvature directions.The analysis focuses on a label-noise-dominated setting with constant-order noise.Source: §3.1 · p. 5

3.2 Optimization methods

• Mini-batch gradient noise separates into iterate-dependent multiplicative noise and persistent label-noise-induced additive noise, which govern stability and the asymptotic noise floor, respectively.Polyak uses the current gradient, whereas Nesterov evaluates it at a look-ahead point.Source: §3.2 · pp. 5–6

3.3 Regime-dependent damping in momentum methods

• For quadratic directions, overdamped Polyak behaves like gradient descent with learning rate η/(1−ρ), whereas underdamped directions oscillate with amplitude decaying as ρ^k/2.The transition occurs at ηλ = (1−√ρ)^2; overdamped decay depends on curvature, but underdamped amplitude does not.Source: §3.3 · pp. 6–7
• Nesterov replaces the momentum factor by ρ(1−ηλ), damping sharper underdamped directions faster while matching Polyak approximately in sufficiently flat directions.The spectrum therefore splits into underdamped heads and overdamped tails for separate risk analysis.Source: §3.3 · p. 7

Definitions

• The critical learning rate is the largest learning rate compatible with risk stability, measured through population risk for SGD, Polyak, and Nesterov.Risk stability is defined for every admissible initialization with finite population risk; the relevant hyperparameters are B for SGD and (B, ρ) for momentum methods.Source: §4 · p. 7

Critical learning-rate scaling

• η_SGD^crit ≍ 1, η_Polyak^crit ≍ min{1, B(1−ρ)}, and η_Nesterov^crit ≍ min{1, B^β(1−ρ)} for sufficiently large B and sufficiently small 1−ρ.The implicit constants are independent of B and ρ.Source: §4 · p. 7

Joint dependence on momentum and batch size

• Increasing ρ toward one reduces the critical learning rate, whereas increasing B enlarges it; Nesterov’s dependence is stronger, scaling as B^β rather than Polyak’s B before order-one ceilings.The stronger Nesterov scaling originates from curvature adaptivity induced by its look-ahead mechanism.Source: §4 · p. 8

Noise feedback and stability

• Risk stability is governed by multiplicative-noise feedback through a renewal kernel with ||K||_1 < 1, while additive label noise acts only as external forcing.Because K is independent of the label-noise scale σ, the stability condition is also independent of σ.Source: §4 · p. 8

Damping regimes

• Increasing momentum produces three regimes: fully overdamped SGD-like dynamics, mixed damping with underdamped transients, and eventual risk instability.The damping threshold 1 − ρ ≂ √η and stability threshold 1 − ρ ≂ η/B^q delimit these regimes.Source: §5 · p. 8
• The curvature cutoff separates flat overdamped directions Iover from sharp underdamped directions Iunder, with overdamped modes always present because the spectrum has arbitrarily small eigenvalues.The high-momentum transition uses ηλ ≂ (1 − ρ)^2.Source: §5 · p. 8

Polyak momentum

• Polyak’s mixed-damping law combines flat-mode power-law decay with a sharp-mode transient P(k) bounded by ρ^k; instability occurs when 1 − ρ ≲ η/B.Increasing ρ accelerates flat-mode learning through the effective rate ηρ but slows sharp-mode relaxation on the timescale set by 1 − ρ.Source: §5 · p. 9
• After its transient decays, Polyak follows the SGD law with η replaced by ηρ, while its admissible effective rate grows to ηρ ≲ B without improving the best data-scaling exponent.This enlarged admissible range increases the critical batch size.Source: §5 · p. 9

Nesterov momentum

• Nesterov matches Polyak in the overdamped regime but its curvature-adaptive look-ahead damps sharp directions more strongly, suppressing underdamped transients and stochastic noise accumulation.The asymptotic signal term remains (ηρk)^−s, while instability occurs when 1 − ρ ≲ η/B^β.Source: §5 · pp. 9–10

• Final-step risk is optimized as batch size B grows with data budget D, under the one-pass constraint T = D/B and stability constraints.The analysis focuses on large-batch scaling rather than fixed batch sizes.Source: §6 · p. 10
• For B = D^b+o(1), r(b) measures optimal data efficiency at batch-size exponent b, with larger r(b) indicating better scaling.The methods are compared through r_sgd(b), r_polyak(b), and r_nesterov(b).Source: §6 · p. 10
• The transition exponents satisfy 0 < b1 < b2 < b3 < 1, where b1 marks Nesterov’s improvement, b2 maximizes Nesterov’s exponent, and b3 is Polyak’s critical exponent.These transitions define the three batch-size regimes in the theorem.Source: §6 · pp. 10–11
• For small batches b ≤ b1, SGD, Polyak, and Nesterov achieve the same optimal data-scaling exponent.The methods remain equally data-efficient through the first transition.Source: §6 · p. 11
• For b1 < b < b3, Nesterov outperforms Polyak, which outperforms SGD, with Nesterov best at b = b2.The ordering is r_nesterov(b) > r_polyak(b) > r_sgd(b).Source: §6 · p. 11
• For ultra-large batches b ≥ b3, Polyak and Nesterov tie and both outperform SGD.Their exponents satisfy r_nesterov(b) = r_polyak(b) > r_sgd(b).Source: §6 · p. 11

Risk stability boundaries

• The measured stability boundaries confirm η_SGD^crit remains order one, while both momentum boundaries scale linearly with 1−ρ before reaching the order-one ceiling.These results use (s, β) = (3, 2), with batch-size and momentum-dependence tests shown in Figure 2.Source: §7 · p. 11, Figure 2 · p. 12

Risk-dynamics scaling laws

• Risk dynamics exhibit an underdamped transient, common (ηρk)^−s decay for Polyak and Nesterov, and distinct noise floors, with Nesterov saturating substantially lower.The transient transition timescale has dominant (1−ρ)^−1 dependence, and the results use fixed (η, ρ, B) with (s, β) = (0.3, 4).Source: §7 · p. 11, Figure 2 · p. 12

Batch-size phase diagram

• At fixed data budget, SGD, Polyak, and Nesterov match at small batches; Polyak remains effective longer, while Nesterov achieves the best large-batch risk before ultra-large-batch deterioration.This confirms Polyak’s enlarged critical batch-size range and Nesterov’s further data-efficiency improvement.Source: §7 · p. 11

Data-scaling exponents across batch-size regimes

• Measured data-scaling exponents match theory: all methods tie at small batch, Nesterov > Polyak > SGD at large batch, and Nesterov = Polyak > SGD at ultra-large batch.Representative scalings are B ≂1, B ≂D^0.4, and B ≂D^0.8.Source: §7 · p. 12, Figure 3 · p. 12

Conclusion

• Polyak preserves SGD’s optimal data-scaling exponent across a substantially wider batch-size range, whereas Nesterov improves the exponent in the large-batch regime before both become signal-limited.At small batch sizes, neither method improves vanilla SGD’s data-scaling exponent; at ultra-large batches, sequential-update limits produce the same signal-limited scaling.Source: §8 · pp. 12–13

Implications for momentum scheduling

• Increasing-momentum schedules can eventually cross stochastic stability boundaries, so their stable duration grows with batch size and differs between Polyak and Nesterov.The theory predicts the stability horizon for classical updates, while a complete theory of time-varying schedules remains beyond the analysis.Source: §8 · p. 13
• Nesterov remains stable for substantially more iterations than Polyak under increasing momentum because its stability horizon scales as B^β rather than B, reaching lower risk before instability.In the PLK regression example with (s, β) = (0.3, 4) and ρ_k = 1 − 1/(k + 1), both methods initially benefit before becoming unstable.Source: §8 · p. 13
• Momentum schedules should jointly maintain 1 − ρ_k ≳ η_k/B_k for Polyak and 1 − ρ_k ≳ η_k/B_k^β for Nesterov.Increasing momentum can be supported by increasing batch size or decreasing learning rate, connecting the stability theory directly to schedule design.Source: §8 · pp. 13–14

Outlook

• Future work should extend the framework to time-varying η_k, B_k, and ρ_k to guide stable, data-efficient schedules instead of relying primarily on empirical tuning.The outlook emphasizes jointly designing learning rate, batch size, and momentum because stability and scaling laws govern their tradeoffs.Source: §8 · p. 14

Use of AI

• AI tools supported manuscript drafting and language editing throughout the study, while authors developed and verified the mathematical arguments.The authors also verified all AI-assisted calculations and estimates.Source: Use of AI · p. 14
• In Sections B–F, AI tools accelerated algebraic calculations and asymptotic estimates.Source: Use of AI · p. 14

A.1 Equivalence to kernel regression

• PLK regression is related to standard kernel regression over the reproducing kernel Hilbert space associated with a positive semidefinite kernel.Under mild conditions, Mercer’s theorem supplies the kernel eigendecomposition.Source: §A.1 · p. 21
• The standard capacity condition assumes eigenvalue decay λ_j ≂ j^-β, with β as the capacity exponent.Source: §A.1 · p. 21
• Target regularity is described through interpolation spaces, and the task-difficulty assumption characterizes the target coefficients.The source regularity parameter s characterizes target regularity.Source: §A.1 · p. 21
• The PLK formulation is the corresponding feature-map representation under Assumption 3.1.Source: §A.1 · p. 21

A.2 Rotational equivariance of SGD and momentum methods

• SGD, Polyak, and Nesterov are invariant under unitary basis changes, so predictions and risks can be analyzed in the eigenbasis of H.This reduction diagonalizes the covariance operator without loss of generality.Source: §A.2 · p. 22

A.3 Gaussian fourth-moment identities

• Gaussian fourth-moment identities and empirical covariance fluctuation formulas provide the operator moment calculations used throughout the risk-stability proofs.The identities apply to Gaussian covariates with positive trace-class covariance operators and follow from Isserlis’ formula.Source: §A.3 · p. 22

A.4 Noise–curvature alignment

• Noise–curvature alignment bounds the mini-batch gradient-noise variance using the curvature-weighted error and the 1/B variance reduction.The bound is obtained from independence of label noise and features together with Cauchy–Schwarz in the H-inner product.Source: §A.4 · p. 23

B.1 Renewal formulation of risk stability

• A renewal risk equation is bounded exactly when its kernel has total mass below one, separating additive label-noise forcing from multiplicative feedback through the optimizer dynamics.The kernel depends on the method and hyperparameters η, ρ, and B, but not on the label-noise variance σ2.Source: §B.1 · p. 24

B.2 Risk stability of SGD

• SGD is risk stable if and only if its coordinate-wise contraction and global renewal-feedback conditions hold, yielding a unique critical learning rate.Under λ_i ≂ i^-β with β > 1, the critical learning rate has the scaling stated in the theorem and corollary.Source: §B.2 · pp. 24, 26

B.3 Risk stability of Polyak

• Polyak momentum is risk stable if and only if its local eigendirection condition and global renewal condition hold, producing a unique critical learning rate governed by B, ρ, λ_max, and Tr(H).The result is obtained by applying the Jury stability criterion to the isolated coordinate blocks and then controlling the globally coupled risk moment.Source: §B.3 · pp. 26–28
• Polyak risk stability is characterized by a unique critical learning rate obtained from a continuous, strictly increasing renewal mass within the local stability interval.The renewal mass vanishes at zero and diverges at the local boundary, yielding a unique critical rate and stated stability interval.Source: §B.3 · p. 29

B.4 Risk stability of Nesterov

• Nesterov momentum is risk stable if and only if its coordinate stability condition holds under a positive trace-class Hessian and nondegenerate label noise.If the condition fails, label-noise forcing excites a nonstable mode and the corresponding coordinate risk is unbounded.Source: §B.4 · pp. 29, 31

nesterov, ρ

• Under λi ≂ i^-β with β > 1, B ≫ 1, and 1 −ρ ≪ 1, Nesterov has a unique global critical learning rate combining an order-one local ceiling with a global bound.The global critical point lies in the second asymptotic regime for large batch size, and the resulting stability interval is stated in the cited result.Source: nesterov, ρ
  · p. 32

C SGD scaling law

• The SGD scaling law applies when η ≲ 1 and for k ≳ 1/η, with the infinite-dimensional formulation obtained by taking model capacity M →∞.The finite-width approximation error M^-sβ vanishes in this limit under the matching noise covariance structure.Source: C SGD scaling law · p. 33

D Polyak scaling law

• The Polyak scaling-law analysis derives a discrete risk recursion, separates overdamped and underdamped modes, treats additive and multiplicative noise, and analyzes signal learning in two damping regimes.Under risk stability, multiplicative noise can be absorbed into the label-noise term up to constant factors.Source: D Polyak scaling law · p. 33

D.1 Signal–noise decomposition and risk recursion

• Polyak risk admits a two-sided constant-factor decomposition into deterministic signal bias, accumulated additive label noise, and accumulated multiplicative gradient noise.The decomposition follows from an exact coordinate-wise impulse-response representation and the curvature-aligned noise covariance, whose mini-batching variance contribution scales as 1/B.Source: §D.1 · pp. 34–36

D.2 Overdamped and underdamped modes decomposition

• Polyak characteristic roots transition at ηλj = (1 −√ρ)^2, asymptotically ηλj ≂ (1 −ρ)^2, separating real-root overdamped from complex-root underdamped modes.In mixed damping, a crossover index Jmax separates underdamped and overdamped eigendirections.Source: §D.2 · p. 36
• The system is fully overdamped when ηλ_max ≲ (1−ρ)^2; otherwise, high-curvature modes are underdamped while flatter modes remain overdamped.The underdamped subset consists of modes with λ_j ≳ (1−ρ)^2/η, and the remaining spectral tail is overdamped.Source: §D.2 · p. 37

D.3 Label noise calculation

• The finite-time label-noise contribution is bounded above and below by constants times its saturated spectral Green-energy expression, establishing the additive noise floor.For each coordinate, the saturated Green energy scales as G_τ(j)^2 = 1 + ρ / ((1−ρ)ηλ_j(2(1+ρ)−ηλ_j)).Source: §D.3 · pp. 37–39

D.4 Uniform boundedness of expected risk

• With σ > 0 and η ≲ min{1, B(1−ρ)}, expected excess risk remains uniformly bounded, allowing multiplicative gradient noise to be absorbed into the additive label-noise floor.Under these conditions, expected excess risk is asymptotically equivalent to deterministic bias plus label noise.Source: §D.4 · pp. 39–40

D.5 Risk dynamics in the fully overdamped regime

• In the fully overdamped regime η ≲ (1−ρ)^2, deterministic bias stays near initialization through early and intermediate stages before decaying polynomially, while label noise contributes K_polyak(τ) ≂ σ^2η/[B(1−ρ)].The resulting scaling law applies under η ≲ min{1, B(1−ρ)} and sufficiently long times k ≳ (1−ρ)/η; expected risk combines the polynomial bias with the noise floor.Source: §D.5 · pp. 40, 42, 43

D.6 Risk dynamics in the mixed-damping regime

• In the mixed-damping regime η ≳ (1−ρ)^2, Polyak’s deterministic bias splits into an overdamped polynomial tail and an underdamped oscillatory transient.The overdamped contribution reaches the terminal polynomial scaling, while destructive interference prevents a strictly positive pointwise lower bound for the underdamped component.Source: §D.6 · pp. 43–45
• Under stability η ≲ min{1, B(1−ρ)}, the mixed-damping excess-risk law combines the deterministic components with a saturated Polyak noise floor K_polyak(τ) ≂ σ^2η/[B(1−ρ)].The scaling theorem applies for k sufficiently large, and multiplicative noise is absorbed into the additive label-noise floor.Source: §D.6 · p. 45

E Nesterov scaling law

• Nesterov’s analysis derives an impulse-response risk recursion, separates signal learning from stochastic terms, and shows its additive label-noise contribution yields a lower noise floor than Polyak.Under risk stability, the multiplicative-noise contribution is absorbed into the label-noise term up to constant factors.Source: E Nesterov scaling law · p. 46, §E.1 · pp. 47, 49

E.2 Overdamped and underdamped modes decomposition

• Nesterov’s characteristic roots transition at ηλ_j ≂ (1−ρ)^2, partitioning modes into overdamped real-root directions and underdamped complex-conjugate directions.The crossover index satisfies ηλ_Jmax ≂ (1−ρ)^2; globally, either all modes are overdamped or high-curvature modes are underdamped while flat modes remain overdamped.Source: §E.2 · p. 50
• The exact damping boundary scales as ηλ_j ≂ (1−ρ)^2, separating underdamped modes j ≤ J_max from overdamped modes j > J_max in the mixed regime.The crossover is obtained from λ_j ≂ j^−β and ηλ_Jmax ≂ (1−ρ)^2; whether it lies below the leading eigenvalue determines the global regime.Source: §E.2 · p. 51

E.3 Label noise calculation

• Nesterov’s saturated label-noise contribution uses the spectral weight λ_j/(λ_j + (1−ρ)/η), reflecting the effective root product ρ(1−ηλ_j) rather than Polyak’s λ_j weighting.Under the stable small-step condition ηλ_1 ≤ c_st, finite-time sums capture the saturated Green energy up to constants, yielding the two-sided label-noise estimate.Source: §E.3 · pp. 51–52

E.4 Uniform boundedness of expected risk

• For σ > 0, Nesterov’s expected excess risk is uniformly bounded for all k when η ≲ 1 and η ≤ cB^β(1−ρ).Under η ≲ min{1, B^β(1−ρ)}, multiplicative gradient noise is bounded by the additive label noise up to a constant factor, so risk is asymptotically equivalent to deterministic bias plus label noise.Source: §E.4 · pp. 53–54

E.5 Risk dynamics in the fully overdamped regime

• In the fully overdamped regime, real characteristic roots eliminate oscillations, with a slow component dominating after an initial fast transient and producing three temporal stages of risk decay.The regime requires ηλ1 ≲ (1−ρ)^2, and the slow-root coefficient is uniformly positive while the fast-root coefficient is of order c0.Source: §E.5 · pp. 54–57

E.6 Risk dynamics in the mixed-damping regime

• In the mixed-damping regime, expected excess risk combines an overdamped polynomial tail with an underdamped oscillatory transient under the stability condition η ≲ min{1, B^β(1−ρ)}.The underdamped contribution is controlled by a piecewise envelope uniformly across time, but destructive interference can make its pointwise lower bound zero.Source: §E.6 · pp. 58, 59, 61

F Optimal data scaling across batch-size regimes

• The optimal-data-scaling analysis imposes a noisy setting and fixed data-budget constraint, prescribing the batch exponent while optimizing the remaining hyperparameters.The analysis uses transition exponents defined earlier in the paper.Source: F Optimal data scaling across… · p. 62

F.1 SGD

• SGD achieves the optimal data-scaling exponent s/(s + 1) only for batch exponents b < b1 = s/(s + 1), making b1 its critical batch exponent.For b ≥ b1, the signal term limits the rate, yielding r_SGD(b) = s(1 − b).Source: §F.1 · p. 62

F.2 Polyak

• Polyak balances signal and additive-noise powers at b + c − m = b1, with r_Polyak(b) = s/(s + 1) for all b < b3.For b1 ≤ b < b3, choosing c = 0 and m = b − b1 enters the mixed-damping regime while preserving the optimal exponent.Source: §F.2 · p. 63
• For b ≥ b3 = (2s + 1)/(2s + 2), Polyak’s optimal exponent becomes r_Polyak(b) = 2s(1 − b), with the signal term dominant.The endpoint at b3 is understood in the supremal sense, represented by the o(1) term.Source: §F.2 · p. 63

F Optimal data scaling across batch-size regimes

• Polyak extends the critical batch size from D^b1 to D^b3 without changing the optimal small-batch exponent s/(s + 1).The analysis parameterizes momentum and learning rate as 1 − ρ ≂ D^−m and η ≂ D^−c under the condition b + m < 1.Source: §F.2 · pp. 63–64

F.3 Nesterov

• The Nesterov analysis imposes b + m < 1, with stability condition βb + c −m ≥ 0 and damping threshold c = 2m.The transient is then smaller than any polynomial power of D, enabling optimization across fully overdamped and mixed-damping regimes.Source: §F.3 · p. 64
• For b1 ≤ b < b2, the branch c ≤ m has the optimal exponent, while for b ≥ b2 the supremal decay exponent is 2s(1 −b).The transient condition defines b2 through b < (2s + 1/β)/(2s + 1 + 1/β).Source: §F.3 · pp. 65–66
• For b1 < b < b2, mixed damping yields exponent s(1 + (β −1)b)/(sβ + 1), exceeding fully overdamped Nesterov's s(1 −b).For b ≥ b2, mixed damping yields 2s(1 −b), also exceeding the fully overdamped exponent.Source: §F.3 · p. 66
• Nesterov matches SGD and Polyak for small batches, improves its data-scaling exponent beyond b1, and reaches its maximal exponent at b2.For b ≥ b2, the exponent becomes 2s(1 −b), matching Polyak in the ultra-large-batch regime.Source: §F.3 · p. 66

G Experimental details and additional results

• For numerical experiments, the infinite-dimensional PLK system is truncated to the first d = 4096 eigendirections; unless otherwise stated, all experiments use d = 4096.Source: G Experimental details and additional… · p. 66

G.1 Risk stability boundaries

• Empirical stability boundaries average the largest stable learning rate over five seeds, using 2,000 iterations and final-20-iteration risk thresholds.Stability-versus-batch scans fit constant SGD, B(1 −ρ) Polyak, and Bβ(1 −ρ) followed by a constant Nesterov plateau.Source: §G.1 · pp. 66–67
• Stability-versus-momentum-damping fits ηcrit = min{L, c(1 −ρ)} for Polyak and Nesterov, while SGD remains constant across damping.Additional stability boundaries are reported for different (s, β) choices in Figure 4.Source: §G.1 · p. 67, Figure 4 · p. 67

G.2 Risk-dynamics scaling laws

• Across additional (s, β) settings, fitted Polyak and Nesterov scaling forms consistently track empirical excess-risk dynamics.These experiments use σ = 1, B = 32, η = 0.03, ρ = 0.98, 5 × 10^5 iterations, and averages over 100 runs.Source: §G.2 · p. 68, Figure 5 · p. 68

G.3 Batch-size phase diagram at a fixed data budget

• Figure 1 (right) reports the minimum E for each method and batch size over the corresponding local hyperparameter search grids.The evaluation uses a fixed data budget D = 214, 50 independent runs, and mean excess risk over the final W iterations.Source: §G.3 · p. 68
• Additional batch-size phase-diagram results are reported across different (s, β) settings in Figure 6.The figure is described as extending the phase-diagram experiments beyond the primary parameter setting.Source: §G.3 · p. 68, Figure 6 · p. 68

G.4 Data-scaling exponents across batch-size regimes

• Data-scaling experiments evaluate one representative batch-size exponent from each of the small-, large-, and ultra-large-batch regimes.For each data budget, the best stable configuration is reported after tuning multiplicative constants around the predicted learning-rate and momentum scalings.Source: §G.4 · p. 69
• In the small-batch regime, all methods use η ≂ D^-s/(s+1), while momentum methods additionally use 1 − ρ ≂ 1.The regime is defined by bsmall = 0.Source: §G.4 · p. 69
• In the large-batch regime, SGD uses η ≂ 1, while momentum methods use η ≂ 1 with momentum scaling around 1 − ρ ≂ D^-(blarge−s/(s+1)).The representative exponent is blarge = b2 − δ, with b2 = 2sβ+1 2sβ+β+1 and small δ > 0.Source: §G.4 · p. 69
• In the ultra-large-batch regime, SGD uses η ≂ 1, while both momentum methods use η ≂ 1 and 1 − ρ ≂ D^-(1−bultra−δ).The choice satisfies b3 < bultra < 1 and requires δ > 0 such that T(1 − ρ) → ∞.Source: §G.4 · p. 69
• Additional data-scaling results across parameter settings are presented in Figures 7 and 8.Figure 7 uses (s, β) = (0.1, 4), while Figure 8 uses (s, β) = (0.3, 2).Source: §G.4 · p. 69, Figure 7 · p. 69, Figure 8 · p. 69

G.5 Delayed instability under increasing momentum

• Delayed-instability experiments isolate multiplicative noise by setting σ = 0 and use η = 0.05, B = 4, and an increasing momentum schedule.For Polyak and Nesterov, the schedule is ρk = 1 − 1/(k + 1); runs last at most 5 × 10^5 iterations over 50 runs, reporting median risk.Source: §G.5 · p. 69