PaperlayerView exact revision on arXiv

Source-linked AI summary

A Solvable Model of Adaptive Learning Rate Rescaling: Acceleration, Stability & Scaling

Itay Lavie, Clarissa Lauditi, Cengiz Pehlevan

arXiv:2610.06701v1stat.MLcond-mat.dis-nncs.LG

TL;DR

The consequences of adaptive update-magnitude rescaling for learning-curve and resource scaling remain unclear. The paper derives a DMFT for normalized SGD in a power-law random-feature model, finding transient acceleration and stability-limited late-time regimes that shape batch- and width-related compute trade-offs.

  • Problem

    How adaptive learning-rate rescaling interacts with spectral learning dynamics and resource scaling remains poorly understood, despite the importance of scaling laws for allocating compute.

    Source: §I · pp. 1–2

  • Method

    The paper derives a dynamical mean-field theory for normalized SGD in a high-dimensional power-law random-feature model, isolating update-magnitude feedback while preserving the SGD direction.

    Source: §I · pp. 1–2

  • Results

    Normalization maps r_SGD to 2r_SGD/(1-r_SGD) for r_SGD<1, with exponential convergence at r_SGD=1 and formal finite-time convergence above it; late-time behavior separates into width-limited, EoSS, and EoS regimes.

    Source: §I · p. 2

  • Takeaways & Limitations

    In EoSS, nearly linear batch scaling can reduce steps to a target loss at almost unchanged compute, while some width-limited spectral regimes allow potential width speedups.

    Source: §I · p. 2, §V · p. 6

Abstract

from arXiv · show

A recurring design principle in modern optimizers is to decouple update magnitude from the raw gradient norm, yet its consequences for learning-curve and resource scaling remain unclear. We isolate this mechanism by studying normalized SGD in a random-feature model with power-law teacher and data covariance. Fixed-norm updates induce an effective learning rate that grows as gradients shrink. We derive a dynamical mean-field theory (DMFT) describing the joint dependence of the loss on training time, model width and batch size. Normalization initially accelerates SGD, mapping the power-law exponent $r_{\rm SGD}<1$ to $2r_{\rm SGD}/(1-r_{\rm SGD})$, with exponential convergence at $r_{\rm SGD}=1$ and formal finite-time convergence for $r_{\rm SGD}>1$. At finite step size, however, the same feedback ultimately breaks the acceleration and leads to marginal stability. The late-time theory yields width-limited, edge-of-stochastic-stability (EoSS), and deterministic edge-of-stability (EoS) regimes. These phases determine when larger batches or wider models reduce serial training time at comparable compute. We quantify in which of these phases increased batch size or width can compensate the excess compute use per step by fewer optimization steps to target loss. Linearized ResNet experiments on CIFAR-5M support the predicted acceleration, breakdown, and resource-scaling trends. Together, these results connect normalization-induced acceleration, EoS effects, and width--batch allocation within a solvable theory.

I. INTRODUCTION

The paper isolates how adaptive update-magnitude rescaling affects convergence, stability, and resource scaling. It derives a DMFT for normalized SGD and studies its predictions alongside linearized ResNet experiments on CIFAR-5M.

Approach

  • The DMFT characterizes loss scaling jointly with training time, model width, batch size, and finite step size in a power-law random-feature model.Norm-SGD isolates the feedback mechanism while preserving the stochastic-gradient direction and fixing the update norm.Source: §I · pp. 1–2

Main results

  • Normalized SGD accelerates the power-law exponent from r_SGD to 2r_SGD/(1-r_SGD) for r_SGD<1, with exponential convergence at r_SGD=1 and formal finite-time convergence above it.At finite step size, growth of the effective learning rate eventually reaches a stability limit and breaks the acceleration.Source: §I · pp. 1–2

Late-time regimes

  • A spectrum-dependent width–batch phase diagram separates finite-width-limited underparameterized models from overparameterized EoSS and EoS regimes.In overparameterized models, a width-dependent critical batch separates stochastic from deterministic edge-of-stability behavior.Source: §I · p. 2

Resource scaling

  • In EoSS, nearly linear batch scaling can reduce steps to a target loss at almost unchanged compute; width-limited regimes can offer potential width speedups in identified spectral regimes.The latter occurs where stationary effective learning rate grows superlinearly with width.Source: §I · p. 2

Experiments and scope

  • Linearized ResNets on CIFAR-5M support the predicted acceleration, crossover toward SGD-like scaling, and batch- and width-efficiency trends in the explored regime.The analysis isolates a mechanism rather than advocating norm-SGD as a practical optimizer.Source: §I · p. 2

A. Related Work

Prior work links neural scaling to spectra and optimizer behavior, while the paper positions normalized SGD as a way to isolate update-magnitude control and study its interaction with batch thresholds and stability.

Theories of Neural Scaling Laws

  • Theoretical neural-scaling accounts connect power-law exponents to kernel spectra, target alignment, random-feature or hierarchical data models, and increasingly to training dynamics and compute-optimal regimes.Empirical scaling laws quantify loss dependence on model size, data, and compute.Source: A. Related Work · p. 2

Optimizers and Scaling

  • Optimizer comparisons find reported gains depend on tuning, model scale, and training horizon, while schedules and preconditioning shape finite-time scaling.Grafting and norm-based views separate update direction from magnitude, motivating the paper's focus on magnitude control with the SGD direction unchanged.Source: A. Related Work · p. 2

Critical Batch Size and Compute Efficiency

  • Critical-batch work studies the trade-off between more computation per update and fewer serial updates, with prior thresholds tied to data size or spectral structure.This paper studies spectrum-dependent departures from linear effective-step scaling and distinguishes batch efficiency from the EoSS–EoS transition.Source: A. Related Work · pp. 2–3

Training Stability and Large Step Size Effects

  • Prior stability studies connect large learning rates to edge-of-stability dynamics, minibatch fluctuations to EoSS, and curvature scaling to learning-rate transfer across widths.The figure summarizes transient convergence regimes and shows the continuous-time prediction ending when the effective learning rate reaches the stability threshold.Source: A. Related Work · p. 3, Figure 1 · p. 3

II. MODEL

The model is an online power-law random-feature setting in which DMFT reduces high-dimensional training dynamics to effective processes for individual spectral modes.

Model

  • The model uses fresh zero-mean inputs, a frozen random projection onto N features, and trainable weights on those features.The covariance eigenvalues follow λ_k=k^-b and target coefficients obey β_k^2=k^-2a, with 2a+b>1 ensuring finite initial loss.Source: §II · p. 3

Proportional limit

  • The analysis takes a proportional limit in dimension, feature count, and batch size at fixed width and batch ratios.The passage defines ν as the width ratio N/d.Source: §II · p. 3

Dynamical mean-field theory

  • DMFT reduces the dynamics to effective stochastic processes for individual spectral modes, whose correlations and responses concentrate to deterministic limits.Equal-time correlations determine observables such as loss and squared gradient norm; response functions encode perturbation propagation across steps.Source: §II · p. 3

III. CONTINUOUS TIME DYNAMICS

In the continuous-time limit, normalization reparameterizes time while preserving gradient-flow update directions, changing temporal loss scaling. The resulting acceleration is transient: at finite step size, stability ultimately limits it.

Continuous-time dynamics

  • As the step size tends to zero, normalized updates remain parallel to gradient flow and become ordinary gradient flow under a reparameterized time whose effective learning rate grows as the gradient norm decays.This changes temporal scaling while leaving width- and data-limited exponents unchanged from gradient flow.Source: §III · pp. 3–4

Stability and late-time regimes

  • At finite step size, gradient decay raises the effective learning rate until finite width, minibatch fluctuations, or deterministic instability cuts off acceleration.The late-time theory distinguishes width-limited behavior, EoSS, and EoS; increasing ν removes the width bottleneck, while increasing α drives the EoSS–EoS transition.Source: Figure 2 · p. 4, §III · p. 4

Figure 2

  • The phase diagram colors the stationary effective learning rate, and its simulations match the DMFT predictions across the displayed regimes.The figure also plots η_eff over time at ν=1, with simulations, DMFT, and stationary predictions distinguished by dots, dashed curves, and horizontal dotted lines.Source: Figure 2 · p. 4, §III · p. 4

Spectral dynamics

  • In the reparameterized clock, residual modes decay exponentially and the unresolved spectral tail determines the loss scaling.The SGD exponent is r_SGD=(2a+b-1)/b; expressing the trajectory in physical time gives the normalized-SGD convergence regimes.Source: §III · p. 4

Transient convergence

  • For a<1/2, normalization accelerates power-law convergence; at a=1/2 convergence is exponential, and for a>1/2 it is formally finite-time.The flow prediction holds until the effective learning rate reaches the leading-mode stability threshold.Source: §III · p. 4

IV. DISCRETE-TIME DYNAMICS: EDGE OF STABILITY AND BREAKDOWN OF THE CONTINUOUS DESCRIPTION

As gradients decay, normalized SGD’s effective learning rate grows, but finite-step dynamics eventually produce a stationary learning rate. The late-time behavior falls into width-limited, EoSS, or EoS phases.

Breakdown of acceleration

  • At finite step size, the effective learning rate grows as the gradient decays until finite width, minibatch fluctuations, or deterministic instability cuts off acceleration.The stationary effective learning rate is then selected by the dynamics, and norm-SGD has the same late-time update rule as SGD with that constant step size.Source: §IV · p. 4

Batch-size dynamics

  • Across a batch-size range, larger batches reach the same target loss in fewer steps, while loss curves approximately collapse when plotted against cumulative effective time.The figure also reports little spread in compute around B = 1,000.Source: §IV · p. 5

Stationary regimes

  • The late-time dynamics have three phases: width-limited approximation error for ν < 1, EoSS under minibatch fluctuations, and EoS when the leading Hessian mode is marginally stable.In the over-parameterized regime, the stability-limited phase is EoSS at small batch and EoS at sufficiently large batch.Source: §IV · p. 5

Stability conditions

  • The EoS condition is equivalent to η∞eff = 2λmax(ν), while the EoSS condition is comparable to minibatch mean-square stability.Here λmax(ν) is the deterministic equivalent of the upper edge of the random-feature Hessian spectrum.Source: §IV · p. 5
  • The EoSS-to-EoS transition occurs at a finite critical batch size αc; beyond it, increasing α brings only a marginal 1/d stability benefit.Source: §IV · p. 5

V. BATCH SIZE AND WIDTH SPEEDUPS

The three phases differ in whether increasing batch size or width can reduce steps to a target loss without substantially increasing compute. The answer depends on the phase and, in Phase 3, on model size relative to data or batch.

Phase 1

  • In Phase 1, increasing batch size does not save enough steps to offset higher per-step compute, whereas increasing width can speed some target losses at roughly equal compute.Source: §V · p. 6

Phase 2

  • In Phase 2, larger batches can yield speedup at roughly equal compute, but increasing width cannot.Source: §V · p. 6

Phase 3

  • In Phase 3, neither larger batches nor wider networks provide approximately equal-compute speedup when ν ≥ 1 or ν ≥ α.Source: §V · p. 6

Why speedups are phase-dependent

  • These limits reflect sublinear effective-step gains: batch gains are only square-root in Phase 1, width gains are slower than square-root in Phase 2, and Phase 3 batch gains vanish while width gains scale at most as √ν.Those gains cannot offset the linear increase in per-step compute.Source: §V · p. 6

A. Batch Speedup at the Edge of Stochastic Stability

On the EoSS branch, larger batches increase the stationary effective learning rate but do not improve batch efficiency throughout the regime. Nearly linear scaling can nevertheless keep the compute overhead small, with the spectrum and dimension shaping where saturation occurs.

Compute–speed trade-off

  • Throughout EoSS, batch-size gains in effective learning rate are sublinear, so larger batches require more compute to accumulate the same effective time, although nearly linear scaling keeps the overhead small.Source: A. Batch Speedup at the Edge of… · p. 6

Approximately equal compute

  • A tolerated compute overhead δ defines a local batch scale αδ ≃ δ/A; δ = 0 corresponds to linear scaling and the same compute and loss with fewer optimization steps.When target-loss progress follows a common effective-time description across batches, αδ characterizes departure from equal compute.Source: A. Batch Speedup at the Edge of… · p. 6

Spectral and dimensional effects

  • For λk = k^-b, flatter spectra spread curvature across more directions and sustain batch-size benefits longer, while steeper spectra saturate sooner.The leading linear approximation also depends on dimension, which determines whether the regime accommodates large physical batches.Source: A. Batch Speedup at the Edge of… · p. 6

Saturation

Effective-time collapse

  • Batch-size loss curves approximately collapse against cumulative effective time, indicating that a fixed target excess loss requires approximately the same effective time across batches.Source: A. Batch Speedup at the Edge of… · p. 7

B. Width Speedup at the Width Limited Phase

Width efficiency can rise on the width-limited branch when widening increases the stationary effective learning rate more than proportionally to compute cost. The size and practical value of this benefit depend on the spectra and the model’s loss floor.

Mechanism

  • On the width-limited branch, adding parameters can raise the stationary effective learning rate more than proportionally to computational cost.Efficiency improves only when the increase in effective learning rate outweighs the added cost per update.Source: B. Width Speedup at the Width Limited… · p. 7
  • Since μ1 is independent of width, reducing Pν increases the stationary effective learning rate, which scales as Pν^-1/2.Source: B. Width Speedup at the Width Limited… · p. 7

Spectral conditions

  • The data spectrum and teacher alignment jointly determine width efficiency: for b > 1 and 2a + b > 1, asymptotic width scalings apply, while for 0 < b ≤ 1 efficiency decreases across the width window for all admissible a.Source: B. Width Speedup at the Width Limited… · pp. 7–8

Linearized-network results

  • In the reported results, the loss decreases with an accelerated power law until the ceiling is reached; afterward, the power law degrades to the standard SGD power law.The width comparison fixes B = 1,000 and η = 0.01.Source: B. Width Speedup at the Width Limited… · p. 8

Width efficiency

  • Width efficiency Qν reaches an interior maximum at ν* < 1; wider networks can reach a target loss faster at approximately the same compute, though a lower loss floor contributes to some examples.Source: B. Width Speedup at the Width Limited… · p. 8

VI. EXPERIMENT WITH REAL DATA AND NETWORKS

Linearized ResNet experiments on CIFAR-5M support the predicted acceleration and its breakdown, while showing how batch size and width affect compute efficiency. The deterministic DMFT tracks the stochastic dynamics well.

VI. EXPERIMENT WITH REAL DATA AND NETWORKS

  • The CIFAR-5M animate-versus-inanimate experiment uses normalized SGD on lazy-regime ResNets, with measured b ≃1.8 and a ≃−0.325.The exponents were extracted from the infinite-width deterministic NTK and are close to measurements for the same architecture in [6].Source: §VI · p. 8
  • With phase 2 (EoSS) inaccessible, the smallest batch is compute-optimal, while narrow networks are more compute-efficient when time-constrained rather than parameter-constrained.The width sweep used w = 4 to w = 256 at fixed B = 1,000; the batch sweep used B = 10 to B = 10,000 at fixed w = 128.Source: §VI · p. 8
  • Deterministic DMFT tracks stochastic dynamics well, and plotted rulers mark accelerated-flow scaling and the breakdown to the standard SGD exponent at marginal stability.The comparison uses the NTK spectrum, target coefficients, and N and B, without information about the eNTK, data, architecture, or task beyond those quantities.Source: §VI · pp. 8–9

VII. OUTLOOK

The outlook connects normalization-induced acceleration and its discrete-time breakdown with batch- and width-dependent compute efficiency, while outlining extensions beyond the kernel regime and Z-transform and DMFT analyses of discrete-time dynamics and steady-state phases. It notes that causal, lower-triangular equations can be solved numerically by stepping through time.

VII. OUTLOOK

  • Normalization accelerates continuous-time convergence, but growing effective learning rates eventually break that acceleration through width-limited noise and approximation error, EoSS, or EoS dynamics.The continuous-time behavior is accelerated power-law, exponential, or formal finite-time convergence depending on the teacher exponent.Source: §VII · p. 9
  • The critical batch ratio αc marks the EoSS–EoS transition, while αδ bounds approximate linear learning-rate and batch scaling that enables equal-compute speedup.The paper quantifies how both scales depend on width and data exponents.Source: §VII · p. 9
  • Linearized ResNets on CIFAR-5M support the predicted acceleration, its breakdown, and batch- and width-dependent compute trends.Extensions beyond the kernel regime require modeling feature evolution, target alignment, and curvature; further directions include schedules, realistic wall-clock costs, compute–parameter trade-offs, and optimizers that modify update direction.Source: §VII · p. 9

Appendix A: A Short Introduction to the Z-transform

  • The Z-transform represents causal sequences as generating functions, converts shifts and convolutions into algebra, and diagonalizes time-translation-invariant operators.Its convergence region for a causal sequence lies outside a circle set by the sequence's growth rate.Source: Appendix A · p. 13, §2 · p. 13, §1 · p. 13

4. Reading dynamics from singularities

  • Z-transform poles encode discrete-time modes: modes decay inside the unit circle, grow outside it, and are marginal on its boundary.The point z = 1 corresponds to long-time or zero-frequency behavior.Source: §1 · p. 13, §4 · p. 14

5. Beyond TTI matrices

  • Double Z-transforms handle general two-time matrices, and angular integration over relative time projects onto their equal-time diagonal.This preserves diagonalization of time-translation-invariant multiplication on either side.Source: §5 · pp. 14–15

6. Use in the steady-state analysis

  • At steady state, Z-transforms reduce time-translation-invariant matrix products and inverses to scalar algebra, while the singular source at ζ = 1 distinguishes the three stability phases.The analysis uses double transforms for two-time correlations and the final-value theorem to obtain equal-time loss and gradient correlations.Source: §6 · p. 15

Appendix B: Deriving the Dynamical Mean-Field Theory Equations

  • The cavity method derives self-consistent response functions for a closed DMFT, whose correlation results track initial-state and current-state propagation and identify an SGD-noise contribution.The SGD-noise term vanishes as α → 1.Source: §1 · p. 16, §2 · p. 18, §3 · p. 20

3. Results

  • Causality makes the equation operators lower triangular, enabling numerical solution by stepping through time.Source: §3 · p. 20

Appendix C: Random Matrix Theory Interpretation

The appendix interprets the response-function Z-transform through random-matrix theory and derives self-consistent conditions for the long-time effective learning rate. These conditions distinguish three stationary phases, including an EoS condition set by the largest eigenvalue.

Appendix C: Random Matrix Theory Interpretation

  • The response-function Z-transform can be interpreted as the companion Stieltjes transform to Λ^1/2 A^T A Λ^1/2, and the EoS condition is equivalent to the sharpness stability condition ηeff = 2/λmax.Here λmax is the largest eigenvalue of the random matrix Λ^1/2 A^T A Λ^1/2.Source: Appendix C · p. 20

Appendix D: Long Time Behavior and Effective Learning Rate Steady State

  • The long-time self-consistency condition admits three stationary phases: EoS, EoSS, and width-limited behavior.The appendix obtains the steady-state effective learning rate from the DMFT equations.Source: Appendix D · pp. 22–24

1. Evaluating the contour integral in ˆK, ˆT

a. Simplifying ˆT

  • The ˆT contour expression is simplified by rewriting its denominator with an established identity and expressing it using gk.The derivation cites identities D.9 and D.6.Source: a. Simplifying ˆT · p. 25

b. Simplifying ˆK

  • Simplifying ˆK requires an additional sixth identity, after which the derivation substitutes the resulting identities into ˆK.The text derives the additional identity from the preceding relations.Source: b. Simplifying ˆK · pp. 25–26

c. Evaluating the contour integral for ˆK at ζ = 1

2. Width Limited Pole limζ→1

  • The physically consistent width-limited pole gives the Phase 1 EoS condition, with η∞eff = 2/λmax.The condition must be solved together with the self-consistent equation for ˆR3.Source: §2 · pp. 28–29

4. Edge of Stochastic Stability Pole limζ→1 ˆK (ζ) −1 = 0

  • The EoSS pole assumption supplies a self-consistency condition for the stationary effective learning rate.The appendix then uses the resulting ηeff to deduce the final loss.Source: §4 · p. 29

Appendix E: Short Derivation of the GF Scaling Law

This section derives normalized-gradient-flow time regimes and examines how effective learning rate and stability shape batch- and width-scaling, including conditions for approximately equal-compute speedup. Across Phases 1–3, increasing batch size or width does not provide such speedup in the regimes analyzed; Phase 3 width gains are at most square-root when N > B, while the under-parameterized case remains unresolved.

Appendix E: Short Derivation of the GF Scaling Law

  • For r_SGD < 1, loss decays exponentially in physical time; for r_SGD > 1, the clock integral converges to a finite time T.The regimes correspond respectively to a < 1/2 and a > 1/2.Source: Appendix E · p. 30

1. Effective learning rate and batch scaling in Phase 2

  • In Phase 2, the effective learning rate grows sublinearly with batch size, so superlinear scaling is impossible and the best case is compute break-even.The quadratic correction to the leading linear term is negative; its coefficient controls the initial slope deficit and the near-linear scaling window.Source: §1 · pp. 31–32

2. Critical batch ratio separating EoSS and EoS

  • The EoSS-to-EoS critical batch ratio is set by the competition between the remaining spectrum and the largest eigenvalue, with the latter dominant for b > 1/2.At b = 1/2 both contributions are of the same order; below it, the remaining spectrum dominates and ν enters at smaller order.Source: §2 · p. 33

3. Scaling with Width at Phase 1

  • For b = 0.5, width efficiency Qν follows ν^-1/2 for a > 1/2, with a logarithmic correction at a = 1/2; DMFT predictions agree with simulations.At b = 1, for a > 1/2, Qν scales as [ν log(1/ν)]^-1/2, and efficiency initially falls before rising toward a peak near ν = 1.Source: §3 · p. 36
  • Width efficiency can eventually be reversed by stability: when the EoS branch is reached below ν = 1, the increasing and decreasing regimes imply an efficiency maximum at an underparameterized width.Its location is determined by the full stationary equations.Source: §3 · p. 38

4. Absence of approximately equal-compute speedup.

  • Approximately equal-compute speedup requires the stationary effective learning rate to grow approximately linearly with the resource being increased.This subsection examines complementary resource directions to those that permit this behavior.Source: §4 · p. 38
  • In Phase 1, doubling batch size yields only a square-root reduction in updates, so total compute rises rather than providing approximately equal-compute speedup.At fixed width and spectrum, the width constraint fixes the effective learning rate, and the small-step asymptotics make the gain square-root rather than approximately linear.Source: §4 · p. 39
  • In Phase 2, width gains are slower than square-root, so doubling width increases the compute required for the same effective time by more than a square-root factor.This conclusion applies while remaining in Phase 2, where the EoSS condition fixes y independently of width.Source: §4 · p. 39
  • In Phase 3, additional batching provides no stationary speedup because the deterministic stability threshold is independent of batch size.The threshold is η_eff = 2/λmax(ν) at fixed width.Source: §4 · p. 39
  • In Phase 3, increasing width yields at most square-root gains in stationary effective learning rate when N > B; the under-parameterized case is not treated.The analysis does not exclude speedups when batch size exceeds total parameter count, but notes this case is far removed from modern practice.Source: §4 · p. 39

Appendix G: Experimental Details

Appendix G specifies linearized CIFAR-5M experiments and reports spectral-fit parameters predicting a transient exponent. Appendix H derives a dimension-free DMFT limit in which the width-limited state preempts EoSS.

Appendix G: Experimental Details

  • On CIFAR-5M, binary animate/inanimate online training uses each of 5 × 10^6 images once; the residual CNN is trained in its initialization linearization with normalized updates.The effective learning rate is η/∥gt∥2.Source: Appendix G · pp. 39–40
  • b = 1.81, 2a + b −1 = 0.17, a = −0.32, and λmax = 0.44 yield rSGD = 0.095 and a predicted transient exponent of 0.21.The predicted exponent is 2rSGD/(1 −rSGD).Source: Appendix G · p. 40
  • The evaluation reports held-out mean-squared error on 4,096 images at 150 logarithmically spaced steps; the DMFT integration is limited to 3,800 steps by memory.The sweep comprises 241 runs with two seeds per cell.Source: Appendix G · p. 40

Appendix H: The Dimension-Free Limit

  • Taking d→∞ before N,B→∞ removes dimension from the kernel setting, leaving N, B, η, and the spectrum; this limit requires b>1 and 2a+b>1.The dimension-free setting corresponds to the paper’s NTK and linearized-network experiments.Source: Appendix H · p. 40

1. The Dimension-Free DMFT Equations

  • Multiplying the proportional-limit relations by d removes dimension and replaces ν,α with N,B; the resulting closure is exact for 1≪N,B≪d.The substitutions include C1→C0/B, νC3→NC3, and dC2→Σ2.Source: §1 · p. 41

2. Steady State and the Absence of the EoSS Phase

  • With infinitely many modes, the width-limited state is reached before EoSS for every N,B,η, leaving only a transition to EoS with a critical batch size set on that branch.The mode sum is no longer bounded by d, which accounts for the main-text ν<1 condition.Source: §2 · p. 41

Related papers

Loading 2610.06701v1…