PaperlayerView exact revision on arXiv

Source-linked AI summary

From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

Yichen Wang, Fanghui Liu, Yudong Chen

arXiv:2609.40148v2cs.LG

TL;DR

Power-law learning curves can depend on learning-rate and batch-size schedules, so their observed exponents are not necessarily fixed properties of a model and its data. The paper analyzes noisy online SGD with linear random features, relating response components to weighted spectral mass and schedule-dependent memory. Controlled LLM experiments report cross-schedule prediction without refitting and fitted exponents near the IM1/LM1 boundary.

  • Problem

    Learning-rate and batch-size schedules can change observed loss, leaving unclear when power-law responses arise and when schedules preserve, change, or destroy them.

    Source: §1 · p. 1

  • Method

    The paper derives an exact conditional Volterra risk equation for noisy online SGD with linear random features and studies its forcing, memory, and joint-schedule response.

    Source: §1 · p. 2, §5.2 · p. 11

  • Results

    Controlled LLM experiments show cross-schedule prediction without refitting, with fitted qK values of 1.017, 0.952, and 0.995 for OpenWebText, FineWeb, and peS2o V2, respectively.

    Source: §5.4 · p. 13, §6 · p. 14

  • Takeaways & Limitations

    The results characterize learning curves as responses jointly shaped by spectrum, target alignment, noise, and schedule, with B_t/η_t controlling noise injection in intrinsic time.

    Source: §6 · p. 13

    The componentwise power-law criteria apply separately to forcing and one-injection memory, not directly to observed loss.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time $T_t=\sum_{s<t}\eta_s$ controls optimization progress, while $r_t=B_t/\eta_t$ controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in $3+3(+2)$ propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched $B/\eta$ paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in $3+3(+2)$ map. The source code for reproducing our experiments is available on https://github.com/yichenblue/spectra-to-schedules-in-pretraining.

1 Introduction

The paper asks when power-law responses arise and whether joint learning-rate and batch-size schedules preserve them in the observed loss. It connects weighted spectral mass, schedule-dependent noise accumulation, and controlled LLM experiments to address those questions.

1.1 Contributions and findings

  • Power laws in forcing and one-injection memory are governed by cumulative weighted spectral mass near zero, not coordinatewise power laws in eigenvalues or target coefficients.The criteria are separate and necessary-and-sufficient for each response component.Source: §1.1 · p. 3
  • The PLRF model yields a 3 + 3(+2) map of long-memory, integrable-memory, and finite-bulk propagation regimes, with finite-width gap formulas and phase-dependent compute-optimal rates.The map classifies forcing–memory propagation rather than assigning universal exponents to observed loss.Source: §1.1 · p. 3
  • Joint schedules accumulate stochastic injections through the memory kernel, yielding boundaries between preserving, changing, or destroying clean power-law decay and a ceiling on late-stage noise reduction.Below the ceiling, the asymptotic gap identifies B_t/η_t but not learning rate and batch size separately.Source: §1.1 · p. 3
  • Controlled nanoGPT experiments report that matched B_t/η_t paths nearly collapse in intrinsic time and that a forcing–memory surrogate transfers across schedules without refitting.The reported fitted response has qK ≈ 1 and qF < 1 across datasets and model scales.Source: §1.1 · p. 3

1.2 Related work

  • The paper extends prior response-law and schedule analyses by jointly varying learning rate and batch size and testing predictions in controlled plain-SGD LLM training.The related-work contrast distinguishes forcing–memory propagation from prior compute-optimal bottleneck classifications.Source: §1.2 · p. 4

2 Problem setup and exact loss dynamics

The setup uses noisy online SGD on fixed linear random features and derives an exact conditional risk recursion. Its forcing and memory terms separate unresolved target error from the persistence of stochastic injections.

2.1 Linear random features trained by online SGD

  • The model uses Gaussian covariates and a linear target with independent zero-mean bounded-variance label noise; a fixed Gaussian random-feature matrix is trained by online SGD.Only the feature coefficients are trained, from zero initialization, with time-varying batch size and learning rate.Source: §2.1 · p. 5
  • Intrinsic time T_t tracks optimization progress, while 1/r_t is the variance injected per unit intrinsic time; exact definitions and recursions hold conditional on the representation.The asymptotic analysis takes width and dimension jointly to infinity under the stated dimension-to-width condition.Source: §2.1 · p. 5

2.2 Forcing, memory, and the exact Volterra equation

  • The exact risk recursion separates forcing, which propagates initial target error without stochastic feedback, from memory, the impulse response of one SGD-variance injection.The spectral filter describes mode-wise error survival and is approximated by exp(−2Tλ) on the long-time scale λ = O(T^−1).Source: §2.2 · pp. 5–6

3 When do frozen spectral dynamics produce power laws?

The paper characterizes when frozen spectral dynamics produce power-law forcing or memory responses, then distinguishes those component laws from the observed loss. Cumulative spectral mass near zero—not individual modes—sets the componentwise power-law criterion.

3.1 From the moving spectral cutoff to a spectral criterion

  • As time grows, modes above the moving cutoff T^−1 have relaxed while slower modes remain, so cumulative weighted spectral mass near zero determines the response.The criterion gives separate if-and-only-if conditions for power-law forcing and one-injection memory under the stated spectral-window conditions.Source: §3.1 · p. 6
  • The finite-width example’s loss crosses from exponentially decaying forcing to a T^−3/4 memory tail, while the memory response and its cumulative mass track that power law.The forcing response and forcing mass decay faster than every inverse power of T.Source: §3.1 · pp. 6–7

3.2 When power laws appear—and when they do not

  • A power-law eigenspectrum does not ensure power-law forcing: exponentially little target energy in slow directions makes forcing decay faster than every inverse power of T.In this construction, the memory decay remains unchanged because it does not depend on the target.Source: §3.2 · p. 7
  • Oscillatory, mass-compensated sparse, and frozen random-amplitude targets can retain the canonical forcing exponent under the stated spectral and parameter conditions.The cited construction uses λ_j = j^−2α, α > 1/4, and 2α + 2β > 1.Source: §3.2 · p. 7
  • A finite spectrum ultimately gives exponential decay, although a long intermediate scaling window may precede that asymptotic regime.Source: §3.2 · p. 7
  • The componentwise criterion does not directly determine the observed loss: memory must first be accumulated over past injections and propagated through feedback.Source: §3.2 · pp. 7–8

4 PLRF propagation regimes: 3 + 3(+2) scaling laws

The PLRF 3+3(+2) map classifies how forcing and one-injection memory propagate, rather than assigning universal exponents to total observed loss. Its response coordinates capture schedule-relevant behavior, while finite-bulk regimes remain distinct because their memory depends explicitly on width.

4 PLRF propagation regimes: 3 + 3(+2) scaling laws

  • The 3+3(+2) map classifies propagation of learnable forcing and one-injection memory, not universal exponents of the observed total risk.The schedule must still accumulate memory over past noise injections to determine the resulting learning curve.Source: §4 · pp. 8–9
  • qF and qK are the decay exponents of forcing and one-injection memory, and remain useful for irregular spectra and targets without coordinatewise α, β power laws.Models with the same response exponents have the same leading schedule behavior.Source: §4 · p. 8
  • The fitted LLM regime lies near α = 1/2, where qK ≈ 1 corresponds to the near-square-root compute-optimal width scaling reported by Chinchilla.The propagation map differs from the 4 + 3 compute-optimal taxonomy because it classifies response propagation before resource optimization.Source: Figure 3 · p. 8, §4 · p. 9
  • Long-memory regimes have 0 < qK < 1 and nonintegrable survival kernels, while integrable-memory regimes have qK > 1; subscripts locate qF relative to qK and 1.Source: §4 · p. 8
  • Finite-bulk regimes remain separate because their memory is width-coupled, so no width-independent decaying exponent qK > 0 represents them.The bulk index grows with width in this regime.Source: §4 · p. 8

5 How joint schedules transform the observed scaling laws

The section tracks schedule effects through intrinsic time and the noise-injection ratio, then studies how forcing and memory shape observed loss. It derives schedule-dependent scaling and resource results and tests a fit-then-transfer surrogate on LLM schedules.

5.1 Schedule response and preserve–change–destroy classification

  • The ratio r(T)=B/η controls stochastic injection, while comparing the resulting noise exponent with the clean exponent classifies scaling as preserved, critical, changed, or destroyed.At the destroy boundary the noisy–clean gap has no power decay, and reducing late-stage noise faces a memory ceiling.Source: §5 · p. 9, §5.1 · p. 10, Figure 4 · p. 10

5.2 PLRF finite-bulk and full-risk asymptotics

  • PLRF risk separates unresolved-signal forcing from accumulated surviving stochastic injections, while finite-bulk regimes retain width–memory coupling rather than forming a third LM/IM branch.The α > 1/4 branch motivates a seven-parameter loss surrogate that can be fitted on one schedule and transferred to WSD without refitting.Source: §5.2 · p. 11

5.3 Optimal ratios and sharp resource rates

  • Most LM/IM regimes share data-optimal rate D^-p/(1+p), but compute-optimal rates differ with width, feature-distortion, and memory bottlenecks; IM1, β < 0, and FB2 are fixed-data exceptions.Table 1 summarizes optimal risk orders across propagation regimes for data and feature-compute budgets.Source: §5.3 · p. 12, Table 1 · p. 12

5.4 LLM experiments

  • Across datasets, fitted qK values are 1.017 on OpenWebText, 0.952 on FineWeb, and 0.995 on peS2o V2; the surrogate transfers from 8-1-1 to WSD without refitting.The fitted values support an effective response coordinate near one.Source: §5.4 · p. 13, Figure 6 · p. 13, Figure 5 · p. 13

6 Conclusion and discussion

The paper treats power-law learning curves as dynamical responses shaped jointly by spectrum, target alignment, noise, and schedule. It connects this account to a 3 + 3(+2) regime map, finite-width and deterministic-equivalent results, and experiments supporting schedule-level predictions.

6 Conclusion and discussion

  • The 3 + 3(+2) map yields an exact conditional finite-bulk gap law, phase-dependent deterministic-equivalent risks and resource-optimal rates, and experimental support for the conditional and DE predictions.LLM experiments also show that the forcing–memory surrogate predicts across schedules without refitting; fitted qK ≈1 corresponds to effective α ≈1/2.Source: §6 · pp. 13–14

Organization and scope of the appendix results

  • Under joint width–time limits, temporal power-law behavior requires the moving filter scale to remain above the finite-width cutoff; at fixed width, the transient eventually decays exponentially.The joint window is 1 ≪ Tm ≪ m^(2α); once T ≫ m^(2α), all positive modes have been resolved.Source: Organization and scope of the appendix… · pp. 20–21
  • Finite-width risk dynamics conditional on frozen features are exact, whereas deterministic equivalents approximate the empirical spectrum and need uniform spectral control for long-time power-law claims.Fixed-horizon validity alone does not establish validity when T = Tm grows.Source: Organization and scope of the appendix… · p. 21

Notation guide

  • The notation distinguishes exact dynamical quantities from their deterministic-equivalent counterparts and marks conditioning on frozen features explicitly.The guide also names PLRF, LM, IM, and finite-bulk abbreviations and distinguishes forcing and memory exponents.Source: Notation guide · pp. 22–23

A Spectral representations and formal power-law criteria

This section develops forcing–memory risk representations and uses spectral conditions to characterize power-law decay in those components and their transfer to observed loss through Volterra dynamics. It derives PLRF exponents from spectral decay and target alignment, and identifies the parameter boundaries and memory-related limits that qualify these laws.

A.1 From exact dynamics to deterministic-equivalent risk

  • Iterating the exact scalar recursion over spectral modes yields the Volterra risk equation, whose feedback denominator sums repeated reinjections of prediction error.Source: §A.1.1 · p. 24

A.1.1 Exact conditional forcing–memory dynamics

  • When κW < 1, nonnegative memory stabilizes feedback with gain at most (1 − κW)^−1, and terminal risk combines frozen-representation error with persistent label noise.Source: §A.1.1 · pp. 24–25

A.1.2 Positive spectral measures for the deterministic equivalent

  • The deterministic equivalent replaces realized spectral projections with positive measures: target weighting yields forcing, while trace and squared-spectrum weighting yield memory.Source: §A.1.2 · pp. 25–26

A.1.3 Assembling the deterministic-equivalent risk

  • Under stable positive feedback, the deterministic-equivalent noisy–clean risk gap stays within a fixed factor of cumulative memory.Source: §A.1.3 · pp. 26–27

A.2 Spectral criteria for power-law learning dynamics

  • Forcing and one-injection memory have power laws exactly when their respective target-weighted and squared-spectrum low-mass tails do; modes of size λ are learned around λ^-1.Source: §A.2 · p. 27
  • The power-law equivalences require a joint width–time spectral window: finite spectra eventually decay exponentially, and the simultaneous width–time uniformity is assumed.Source: §A.2 · pp. 27, 29
  • For qK < 1 cumulative memory grows as (η/B)T^(1−qK)LK(T), whereas for qK > 1 it converges with a remaining tail of the same order; qK = 1 is the dividing line.At equality, a fixed infinite spectrum has a borderline integral.Source: §A.2 · pp. 29–30
  • Stable feedback transfers the clean floor and forcing, and noisy cumulative memory, to observed loss, with renewal adding a kernel-sized term.At a terminal plateau, centered convergence additionally requires uniform terminal-tail control and quantitatively noncancelling positive post-feedback responses; the result covers exact and deterministic-equivalent losses.Source: §A.2 · p. 30
  • Under constant schedules, stable feedback transfers component tails to observed loss: clean loss scales with F0 + F>0, while noisy loss adds σ^2 St.The exact conditional loss and its deterministic equivalent obey the same positive Volterra system.Source: §A.2 · pp. 30–31
  • Renewal feedback adds a renewal contribution to the same loss comparisons, and if F>0(t, m) tends to zero, a fixed-width terminal plateau exists.Source: §A.2 · p. 31
  • For PLRF with 2α + 2β > 1, finite target energy gives the forcing exponent qF = (2α + 2β −1)/(2α).Target alignment affects the forcing exponent.Source: §A.2 · p. 32
  • For PLRF with α > 1/4, the memory exponent is qK = 2 −1/(2α), depending only on spectral decay.Source: §A.2 · p. 32
  • The PLRF exponents arise from a moving spectral cutoff: at intrinsic time T = ηt, critical modes satisfy qη(λ)t = e^-2Tλ(1 + o(1)).The corresponding contributions are unlearned target error and one surviving variance injection.Source: §A.2 · p. 32
  • As 2α + 2β approaches 1 from above, qF approaches zero; at α = 1/4, the LM/IM memory formula gives way to FB.Source: §A.2 · p. 33
  • For α < 1/4, the exact conditional noisy–clean gap is closed without a target-forcing comparison.Source: §A.2 · p. 33
  • For α < 1/4, explicit DE total-risk asymptotics retain the forcing range; for α > 1/4, DE kernel transfer is proved, with that restriction applying to the explicit PLRF forcing profile.Source: §A.2 · p. 33

B Proofs of the spectral power-law criteria

This section proves spectral power-law criteria and develops supporting results for joint learning-rate and batch-size schedules. It connects low-spectrum mass to forcing and memory decay and observed loss, and establishes deterministic-equivalent and stability results, propagation regimes, and finite-bulk spectral, kernel, and noisy–clean gap results.

B.1 Positive spectral representation of the deterministic equivalent

  • The deterministic-equivalent resolvent defines positive probability spectral measures supported on the nonnegative real axis with compact support.Its expansion at infinity supplies the stated moment identities.Source: §B.1 · pp. 33–34

B.2 Uniform Tauberian transfer for moving spectra

  • A uniform Tauberian equivalence links power-law response decay to cumulative weighted spectral mass scaling near the moving cutoff λ ≍ T^-1.Uniform spectral control and a compatible training filter ensure mass outside that window does not alter the filtered response.Source: §B.2 · pp. 34–36

B.3 Spectral criteria for forcing and memory

  • The forcing and one-injection memory criteria follow from parallel applications of the Tauberian theorem to their respective weighted spectral measures.Both use the same constant-schedule limit and effective training filter, with distinct measures and normalizations.Source: §B.3 · p. 36

B.4 Accumulating memory over training

  • Cumulative-memory asymptotics follow from exact geometric summation of the empirical kernel and rescaling under the stated cutoff, endpoint-envelope, and stability hypotheses.The proof also obtains a direct terminal-tail criterion and extends the argument to deterministic-equivalent measures.Source: §B.4 · pp. 37–38

B.5 From forcing and memory asymptotics to observed loss under stable feedback

  • Stable feedback preserves the asymptotic order of forcing and memory in observed loss, with positivity yielding a comparison for noisy loss as well.When forcing vanishes, the terminal response is characterized by the limiting memory; the deviation from it is a sum of nonnegative response terms.Source: §B.5 · pp. 39–40

C Joint learning-rate and batch-size schedules

  • The schedule analysis derives exact conditional risk recursions, time-inhomogeneous deterministic equivalents, spectral-regime asymptotics, compute rates, and integer schedules attaining the rates.The proof roadmap places each result alongside its corresponding derivation.Source: C Joint learning-rate and batch-size… · p. 40

C.1 Exact risk dynamics under joint schedules

  • Intrinsic time controls signal learning and B_t/η_t controls noise per intrinsic-time increment, while the exact risk dynamics retain individual schedule steps.Summing the modal recursions yields an exact conditional risk equation without requiring stability or asymptotic spectral assumptions.Source: §C.1 · pp. 40–41

C.2 Time-inhomogeneous PLRF deterministic equivalent

  • For deterministic joint schedules, the PLRF resolvent-DE recursion has a unique nonnegative solution at each finite horizon, and its exact mode-decay factors depend on the full (η, B) factorization.The factors therefore are not determined by intrinsic time alone.Source: §C.2 · pp. 41–42

C.3 Stable schedules and spectral-mode decay

  • Under joint-schedule stability and regularity, individual spectral modes decay exponentially in elapsed intrinsic time; in the fine-mesh limit, that time determines decay, while finite-step decay depends on individual steps.Row stability keeps the total past-risk feedback uniformly below one, controlling its amplification.Source: §C.3 · pp. 43–44

C.4 PLRF regimes and admissible schedule families

  • The PLRF-specific taxonomy comprises eight open propagation subregimes across finite-bulk, long-memory, and integrable-memory families, excluding stated propagation boundaries.The excluded boundaries include α ∈ {1/4, 1/2}, β ∈ {1/2, α}, and 2α + 2β = 1.Source: §C.4 · p. 44

C.5 Finite-bulk empirical spectrum, kernel, and gap

  • A spectral band containing Θ(m) empirical eigenvalues at scale m^-2α yields a target-independent finite-bulk kernel estimate for FB1 and FB2 when T_t = o(m^2α).The kernel result holds under the stated common schedule margins and applies to deterministic, sample-independent schedules.Source: §C.5 · pp. 45–46
  • In the finite-bulk regime, the exact noisy–clean risk gap is controlled uniformly over eligible schedules and finite-energy targets, with a relative refinement when T_t = o(m^2α).The result applies to both finite-bulk target subregimes because the clean forcing cancels from the gap.Source: §C.5 · pp. 46–47

C.6 Long-memory and integrable-memory deterministic-equivalent risk asymptotics

This section develops deterministic-equivalent risk asymptotics for long- and integrable-memory and finite-bulk regimes under joint schedules, including a conditional high-source extension and conditions for transferring results to realized SGD. It also identifies a width-scale spectral band behind finite-bulk behavior and reports that online SGD tracks the finite-width Volterra recursion and its deterministic-equivalent reduction.

C.6 Long-memory and integrable-memory deterministic-equivalent risk asymptotics

  • For α > 1/4, α ≠ 1/2, the DE memory filter has uniform LM/IM asymptotics, with schedule dependence through intrinsic time and η_s^2/B_s.The α = 1/2 boundary is excluded because it would instead yield a logarithm.Source: §C.6 · pp. 47, 49, 53
  • In the six open LM/IM subregimes, joint-schedule DE forcing and risk are uniformly comparable to explicit asymptotics through T_t ≍ m^(2α).The class-uniform risk bounds apply for α > 1/4, α ≠ 1/2; the comparison constants are uniform over the schedule family, widths, and horizons.Source: §C.6 · p. 54

C.7 Finite-bulk deterministic-equivalent risk asymptotics

  • In finite-bulk regimes, each of order m coordinates contributes fixed positive mass to the same width-scale spectral band.This band underlies the DE kernel and noisy–clean gap; the section states that the gap and total-risk asymptotic need no target-forcing assumption for α < 1/4 and β < 1 + 2α.Source: §C.7 · pp. 54–55
  • The finite-bulk results bound the DE kernel and noisy–clean risk gap, while canonical PLRF target forcing is controlled where the central-contour remainder is negligible.This forcing control closes the total-risk asymptotic without an additional finite-bulk forcing assumption.Source: §C.7 · pp. 55–56
  • Finite-bulk DE risk asymptotics hold in the stated FB1 and FB2 source ranges, with the high-source FB2 pure-power extension conditional on residual forcing control.The extension covers 1 ≪ T_t = o(m^(2α)) for β > 1 + 2α; the boundary β = 1 + 2α is excluded.Source: §C.7 · pp. 56–57

C.8 Conditional transfer from DE risk asymptotics to realized SGD

  • Under fixed-sequence empirical-to-DE transfer, realized SGD risk is comparable to DE risk; class-uniform transfer also makes the exact conditional and DE resource infima comparable.The uniform claim requires transfer to hold simultaneously over the admissible resource-bounded family.Source: §C.8 · p. 58
  • Across the displayed LM, IM, and FB systems, online SGD closely tracks both the exact conditional finite-width Volterra recursion and its resolvent-DE reduction.The passage points to Figures 7 and 8 for this empirical comparison.Source: §C.8 · p. 58

C.9 Optimal ratio and schedule realizations

This section derives the optimal noise-injection ratio under a data budget and characterizes continuum and integer SGD schedules that realize it, including phase-dependent compute-rate qualifications. It also gives fixed-data model-width and intrinsic-time choices for the stated label-noise objectives and specifies conditions for schedule paths and spectral-tail transfer to two-time memory.

C.9 Optimal ratio and schedule realizations

  • The fixed-horizon optimum is proportional to √wT,m, and with ratio bounds it is clipped; feasibility requires r−T ≤ D ≤ r+T.Under the unconstrained positive-control problem, the minimizer is unique up to null sets; at either resource endpoint the bounded optimizer is constant.Source: §C.9 · p. 60
  • The same optimal ratio has fixed-learning-rate and fixed-batch continuum realizations, both processing D samples and reaching intrinsic time T; on an unclipped terminal segment, the schedule decay exponent is 4α−1.Clipping corresponds to learning-rate or batch plateaus, depending on the factorization.Source: §C.9 · pp. 61, 62, 65
  • Integer unit-batch realizations attain the proved subregime orders, and no admissible schedule improves them.The deterministic quantile construction uses exactly D samples and reaches intrinsic time T; the stated schedules satisfy peak and bounded-batch conditions at the relevant horizons and widths.Source: §C.9 · pp. 62–63
  • At α = 1 the IM2–IM3 compute expressions agree at f^−1/2; for α > 1, reaching T ≍ m^2α is source-window saturation, while the IM1 lower-ratio floor is active when β < 0.At β = 0 neighboring exponents agree, but the open-branch statements exclude the crossover; the high-source FB2 rate retains its conditional qualification.Source: §C.9 · pp. 62–63
  • For α > 1/2, β > 1/2, and p = 2α + 2β −1, the IM2–IM3 compute optimum scales as F^−p/(1+p+2β) for 1/2 < α ≤ 1 and F^−α/(1+α) for α > 1.At α = 1, the rate is F^−1/2 with nonunique allocations; for α > 1, the optimizer reaches the source endpoint.Source: §C.9 · p. 66
  • At fixed data, (IM1, β > 0), FB1, LM1, and LM2 match with m ≍ D^(1/(1+p)) and T ≍ D^(2α/(1+p)), while IM2 and IM3 match with m ≍ D^(2β/(1+p)) and the same T.Because σ2 > 0 is fixed and clean forcing is bounded, the upper forcing scale has the same order as these label-noise objectives; all upper constructions use Bs ≡ 1, so batch integrality and bounded-batch propagation are exact.Source: §C.9 · p. 67

D Sharp transfer for regularly varying joint schedules

  • Schedule regularity allows width-dependent slow variation but excludes hidden oscillations or bursts, so polynomial width amplitudes must be tracked separately.The appendix frames its results for fixed infinite-spectrum systems or width–time triangular sequences, with uniform statements using bounds independent of width and horizon.Source: D Sharp transfer for regularly varying… · p. 68

D.1 From the empirical spectral tail to two-time memory

  • Under the stated spectral-tail and global Potter/no-escape conditions, plus a trace bound, empirical spectral tails yield two-time memory envelopes; for PLRF with d/m → c > 1 and 1/4 < α < 1/2, the endpoint trace is tight.The endpoint trace bound excludes polynomial endpoint blowup.Source: §D.1 · pp. 70–71

D.2 Examples and counterexamples for the transfer assumptions

This section uses examples and counterexamples to show how cumulative low-spectrum weighted mass governs component power laws and when transfer assumptions succeed or fail, including failures from spectral gaps, lacunarity, and width-dependent mass. The constructions also distinguish stable long memory from row stability and show that finite-width power laws hold only before a cutoff.

Examples and counterexamples

  • Stretched-exponential spectra that are not regularly varying in index can still yield component power laws with logarithmic corrections on a strict intermediate window.The condition γ < 1 supports the integral approximation; at γ = 1, geometric lacunarity can introduce log-periodic corrections and destroy regular variation.Source: §D.2 · pp. 71–73
  • If cumulative target amplitudes satisfy Cn/n → 1, irregular target coordinates produce the same leading forcing tail power and constant as the canonical target.The constructions include alternating, sparse mass-compensated, and frozen random amplitudes that violate every fixed power envelope.Source: §D.2 · p. 74
  • For a power-law spectrum with faster-than-polynomial forcing, the exact learnable forcing is smaller than every inverse power of T, while the memory response is target-independent.This result is stated along joint limits with T → ∞ and T = o(m^(2α+1)).Source: §D.2 · pp. 74–76
  • Two-time long memory does not imply constant-ratio row stability; an additional shrinking η/B or a raised ratio path is needed.The example's tails satisfy the uniform Potter/no-escape envelope, but constant schedules with η/B bounded away from zero lack a row-stability margin uniform in width.Source: §D.2 · pp. 76–77
  • A non-power spectrum can satisfy stable no-escape for both branches, with pointwise and row-stability margins uniform in width.The example uses eigenvalues and target weights proportional to e^(-√j), with a strict window on log T relative to √m.Source: §D.2 · p. 77
  • Three conditioned systems expose distinct transfer failures: a spectral gap yields exponential decay, geometric lacunarity breaks regular variation, and a moving packet changes the forcing-transform constant despite fixed-cutoff power-law behavior.The packet violates the no-escape envelope at its own scale.Source: §D.2 · pp. 77–79

D.2 Examples and counterexamples for the transfer assumptions

  • Pointwise contraction requires x < 2B/(B + 1), but row stability requires the stricter x < 2B/(B + 2); despite mode contraction, noise can make the exact second moment diverge.Divergence occurs whenever Rσ,0 + σ2 > 0.Source: §D.2 · p. 79
  • At fixed width, all positive modes eventually relax exponentially, so the reported power-law behavior applies only within the displayed pre-cutoff width–time window.The finite-width examples use minibatch-SGD trajectories on fresh Gaussian covariates and compare population excess risk with finite-spectrum forcing and one-injection memory.Source: §D.2 · p. 80
  • For the mass-compensated sparse target, all four risk curves decrease overall as power laws on the resolved window, but geometric spacing prevents the last curve from having a pure-power asymptotic.The displayed forms include a bounded, nonconstant, log-periodic factor.Source: §D.2 · p. 80
  • The rapid-target boundary has F(T) = exp{−Θ(T^5/9)} and R(T) ≍ T^−3/4, whereas the ideal gapped construction has R(T) = O(e^−cT).Removing slow-direction target energy leaves the target-independent memory tail; removing the low-spectrum mechanism yields exponential risk.Source: §D.2 · p. 80
  • Cumulative weighted spectral masses, rather than pointwise spectrum or target form alone, determine long-time curve shape; gaps, lacunarity, and width-shifting mass can defeat transfer assumptions.Pointwise contraction controls individual modes, while row stability controls accumulated feedback.Source: §D.2 · pp. 79–80

D.3 Long-memory schedule transfer with feedback

For long-memory kernels, the noisy–clean gap comes from schedule-dependent injections propagated through memory. Its exponent depends on the schedule-to-memory interaction, with stable branches constrained by feedback and by a boundary below which the row bound fails.

D.3 Long-memory schedule transfer with feedback

  • The schedule-induced noisy–clean gap is formed by accumulating stochastic injections through the memory kernel.The section’s proof separates the injection history into early, bulk, and recent regions to establish the transfer results.Source: §D.3 · pp. 80, 84
  • Below the memory ceiling, if either the direct injection term or the exact gap has a positive-power decay exponent q < qK, their difference is lower order.The feedback contribution is asymptotically negligible relative to the direct injection term under the stated kernel and stability assumptions.Source: §D.3 · p. 83
  • When ϑ ≥ 1, every fixed-system branch has noise exponent qK, although at ϑ = 1 the slowly varying factor depends on whether cumulative injection V converges.For ϑ = 1 with V diverging, the gap tracks the cumulative injection times the memory tail; when V is finite, the gap follows the memory tail.Source: §D.3 · pp. 83–84
  • For 1 − qK < ϑ < 1, the schedule-induced gap has exponent q = ϑ + qK − 1, below the memory exponent qK.At ϑ = 1 − qK, the gap is slowly varying and can dominate positive-power clean decay; below that boundary the row mass grows, violating the stability condition.Source: §D.3 · pp. 84–85

D.4 Integrable-memory schedule transfer

With integrable memory, the schedule-induced gap reflects both recent injections and older injections filtered by the kernel tail. Under the stated all-age and stability controls, its exponent is capped by the memory exponent, and the total loss exponent is further capped by clean decay.

D.4 Integrable-memory schedule transfer

  • For qK > 1, recent injections see total memory mass, whereas early injections see the tail of the memory profile.These two endpoints yield the schedule-versus-kernel exponent boundary in the integrable-memory regime.Source: §D.4 · pp. 85, 87
  • The direct continuum response under integrable memory has power exponent min{ϑ, qK}.The response is controlled by the recent-injection endpoint or the old-injection kernel-tail endpoint; at ϑ = 0 it has zero power exponent, while ϑ < 0 yields growth.Source: §D.4 · pp. 85–86
  • The exact discrete result requires all-age kernel control, bounded intrinsic mesh, and stability; triangular sequences additionally need the stated normalization conditions.For the PLRF integrable-memory regime, applying the exact finite-width result also requires empirical-to-DE transfer and empirical row stability.Source: §D.4 · pp. 86, 88
  • For ϑ > 0, the exact noisy–clean gap has exponent min{ϑ, qK}: recent injections dominate below qK, old injections above it, and both contribute at equality.At ϑ = 0 there is no positive power exponent; ϑ < 0 makes the row mass grow and violates the stability bound under the stated normalization.Source: §D.4 · p. 87
  • If the centered clean loss has exponent q0 > 0, the total loss exponent is min{q0, qN(ϑ)}.This follows from positivity of the clean loss and noisy–clean gap.Source: §D.4 · p. 87

D.5 From the exact noisy–clean gap back to B/η

In a fixed limiting system, the exact gap can identify the asymptotic schedule ratio below the memory ceiling, under regularity and monotonicity conditions. The converse is regime-qualified: identification fails at the ceiling and need not extend to triangular sequences without uniform output behavior.

D.5 From the exact noisy–clean gap back to B/η

  • The discrete-to-continuous comparison uses output asymptotics to control recent schedule height, then compares older discrete cells with the continuum integral.The result assumes eventual monotonicity of the interpolated inverse schedule ratio.Source: §D.5 · p. 88
  • At the ceiling q = qK, gap-to-ratio identification fails because distinct injection tails can produce the same asymptotic gap.The displayed family has unit total injection but distinct tails while its response remains asymptotic to σ2k(T).Source: §D.5 · p. 90
  • For 0 < q < qK, the exact gap decays as T_t^-q L(T_t) if and only if 1/r(T) scales as T^(qK−1−q) L(T), up to the stated beta-function constant.Here r is the schedule ratio B/η; the equivalence is for a fixed limiting sequence with an eventually nonincreasing inverse ratio.Source: §D.5 · p. 90
  • The converse requires a fixed limit and eventual monotonicity of 1/r; sparse bounded spikes can disrupt pointwise regular variation without changing the gap at the target scale.For triangular identification, locally uniform output behavior at all constant-factor subhorizons is needed.Source: §D.5 · pp. 90–91

D.6 Full preserve–change–destroy classification

This section classifies how schedules and memory shape preserve–change–destroy regimes, translating intrinsic-time behavior into iteration-time exponents and setting up constant-schedule compute analysis. It also bounds the discrete infinite-width fixed-noise optimum on the stated strict parameter region using a positive Volterra resolvent.

D.6 Full preserve–change–destroy classification

  • At the long-memory edge θ = 1 − qK and integrable-memory edge θ = 0, positive-power convergence to Rapp is destroyed; below either edge, row stability fails.At the long-memory edge the gap has a slow lower bound, while below it polynomial growth of row mass contradicts stability; recent injection or growing row mass gives the corresponding integrable-memory obstruction.Source: §D.6 · pp. 91–93
  • In the long-memory interior, the noisy exponent is min{q0, θ + qK − 1}: schedules change the clean exponent below q0 and preserve it above q0, while equality preserves the exponent but may renormalize its slow factor.For θ ≥ 1, the centered total exponent is min{q0, qK}; thus qK is the memory ceiling, and θ = 1 can add a logarithmic factor without changing the exponent.Source: §D.6 · p. 92
  • When q0 < qK, distinct noise-gap exponents can yield the same leading clean and total loss, so loss alone identifies schedule exponents only in the noise-dominated range q < q0 and requires q < qK for exponent-level identification.At the memory ceiling, different integrable injection tails still produce stable gaps with exponent qK.Source: §D.6 · p. 93

D.7 Polynomial schedules in iteration time

  • For polynomial iteration-time schedules, the long-memory interior gap exponent becomes (1 − ℓ)(θ + qK − 1), while θ > 1 saturates at (1 − ℓ)qK.At θ = 1 the long-memory ceiling carries a logarithmic correction; in integrable memory, ℓ = b = 0 gives no positive-power gap decay.Source: §D.7 · p. 95

E Constant learning-rate and batch-size schedules

  • Within the analyzed source and finite-time windows, label noise shifts compute toward wider models and fewer iterations.The appendix analyzes constant learning-rate and batch-size schedules under a feature-compute budget and compares clean and noisy optima.Source: E Constant learning-rate and… · p. 96

E.1 Scaling inputs and optimization setup

  • The constant-schedule setup recovers a 4 + 3 clean-risk decomposition, combining unlearned aligned-target error, null-space error, finite-width spectral distortion, and aligned spectral memory from transient SGD noise.The canonical PLRF forcing and kernel estimates establish this decomposition under the stated parameter and stability conditions.Source: §E.1 · pp. 96–97

E.2 Discrete infinite-width fixed-noise optimum

  • For trace-class parameters, the next subsection introduces exact integer-time objects at infinite width before optimizing the fixed-noise schedule.The supplied passage establishes the setup, not the resulting optimum.Source: §E.2 · p. 98
  • The discrete trace-class fixed-noise optimum is uniformly of order σ^2 on the stated strict parameter region.The proposition concerns fixed σ^2 > 0 and parameters restricted to a strict region.Source: §E.2 · p. 99
  • The positive Volterra resolvent converges to σ^2κ∞/(1 − κ∞), bounding the integer-time infimum from above.The passage uses the kernel geometric series to obtain κ∞.Source: §E.2 · p. 99

E.3 Source-window rates and trace-class plateau

The fixed-noise analysis distinguishes source-window optimization below trace class from optimization over a fixed finite-time window above trace class. Its results describe finite-time optima and centered large-width corrections, not noise improving uncentered risk.

E.3 Source-window rates and trace-class plateau

  • With fixed σ2 > 0, the reported exponents apply under open-PLRF-subregime and branch-specific assumptions, including restrictions on α, β, and the optimization window.For α < 1/2 the optimization is over Msrc(f); for α > 1/2 it is over U.Source: §E.3 · p. 100
  • In the three integrable-memory subregimes, the optimal iteration remains finite; within the source window, five FB–LM subregimes balance Fpp ≍σ2St.Figure 12 visualizes the finite-time stopping mechanism using finite-width deterministic-equivalent Volterra dynamics.Source: §E.3 · p. 100, Figure 12 · p. 101
  • The larger exponent in 4+3 Phases Ia, II, and III describes a large-width DE correction around a finite-time local optimum, not improved uncentered risk.Clean and noisy losses are centered at different limits.Source: §E.3 · p. 102
  • Above trace class, the optimizer stops at finite time; below trace class, width and iteration count can diverge while noise favors earlier stopping.The theorem distinguishes optimization over a fixed finite-time window above trace class from a source-window-restricted optimum below it.Source: §E.3 · pp. 99, 100, 102

E.4 Proof of Theorem E.4

This section establishes distinct optimization outcomes below and above the trace-class boundary and derives qualified clean-to-noisy crossover scales. It also tests how intrinsic time, matched learning-rate/batch-size factorizations, and a forcing–memory surrogate organize and track loss across schedules.

E.4 Proof of Theorem E.4

  • Below trace class, the source-window optima tend to zero; above it, the finite-time optimum converges to a strictly positive noise plateau.The proof establishes sub-trace corners within the source window and a positive stable-noise resolvent for the trace-class plateau.Source: §E.4 · pp. 104–105

E.5 Clean-to-noisy crossover scales

  • The clean-to-noisy crossover scales are order predictions whose branches depend on the regime; the IM1 weak-noise prediction requires β > 0.For β ≤ 0, the matching argument gives no diverging weak-noise minimizer; these scales are not sharp transition points, and locations may depend on model parameters.Source: §E.5 · pp. 106–107

F End-to-end language-model experiments

  • Three experiments test ratio-path separation, factorization collapse, and cross-schedule prediction, while a Muon experiment serves as an optimizer-specific external-validity check.The forcing-memory surrogate is fit on one fixed-batch 8-1-1 trajectory and used without refitting to predict an alternative factorization and unseen WSD schedules.Source: F End-to-end language-model experiments · pp. 108–109

F.1 Testing ratio-path effects at matched intrinsic time

  • At matched intrinsic time, larger B/η growth yields lower validation loss, with diminishing improvement at large growth rates.The single-seed experiment measures total validation cross-entropy and does not identify the LM/IM boundary or verify the predicted asymptotic exponent.Source: §F.1 · p. 109, Figure 15 · p. 110

F.2 Testing factorization invariance at matched ratio paths

  • The factorization experiment compares fixed-batch and fixed-learning-rate realizations of the same ratio paths, starting from matched checkpoints and using the same training tape.It tests whether validation loss is governed mainly by intrinsic time and the ratio path rather than by the particular learning-rate/batch-size factorization.Source: §F.2 · p. 110
  • At each completed grid cell, the plain-SGD realizations consume the same contexts, advance intrinsic time equally, and match B/η = 3200g.Each uses 16g training contexts from the same contiguous tape segment, with ∆T = 0.005; optimizer-update counts can differ.Source: §F.2 · pp. 110–111
  • Paired learning-rate and batch-size factorizations nearly coincide in intrinsic time but follow different trajectories by optimizer step.This holds for both the WSD and 8-1-1 ratio paths; the comparison uses plain SGD and hybrid Muon.Source: §F.2 · p. 111, Figure 16 · p. 111
  • A forcing–memory surrogate fitted only to fixed-batch 8-1-1 tracks the alternative factorization and predicts both WSD trajectories without refitting.Its fitted exponents are qK = 1.017 and qF = 0.374, placing the effective memory response close to the qK = 1 boundary.Source: Figure 17 · p. 112

F.3 A forcing-memory surrogate predicts unseen schedules

A forcing–memory surrogate adapted to joint schedules predicts held-out learning-rate/batch-size schedules without refitting. Across datasets and model scales, fitted memory exponents remain near qK = 1, supporting this as an effective response coordinate.

F.3 A forcing-memory surrogate predicts unseen schedules

  • The joint-schedule surrogate retains an intrinsic-time clean-plus-memory structure but makes stochastic injection depend directly on r(T) = B(T)/η(T).Its seven parameters are fitted on one schedule and then frozen for transfer to unseen schedules.Source: §F.3 · p. 112
  • The surrogate replaces fixed-batch FSL parameters with a forcing–kernel parameterization, where qF is the clean exponent and γ = qK − 1 for a pure power-law kernel.The ratio form also covers fixed-learning-rate, varying-batch schedules.Source: §F.3 · p. 112
  • Across OpenWebText, FineWeb, and peS2o, fitted qK values were 1.017, 0.952, and 0.995, and the frozen surrogate predicted held-out schedules without refitting.The three qK estimates lie within 0.05 of one.Source: §F.3 · pp. 112–114
  • With all seven parameters frozen, the unrestricted 300M fit predicts the alternative 8-1-1 factorization and both WSD trajectories, with qK = 0.989 and qF = 0.291.This supports qK = 1 as an effective finite-window coordinate, not as an independently identified asymptotic memory exponent.Source: §F.3 · p. 113, Figure 18 · p. 113
Loading 2609.40148v2…