PaperlayerView exact revision on arXiv

Source-linked AI summary

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Shuchen Zhu, Yuxin Fang, Mingze Wang, Kun Yuan

arXiv:2608.28442v1cs.LG

TL;DR

Progress along flat directions is central to final loss reduction, but amplifying learning rates there can also amplify stochastic noise. The paper proposes curvature-conditioned multiscale momentum with sphere constraints and reports that MuonM consistently achieves lower terminal loss than Muon across diverse pretraining settings.

  • Problem

    Learning-rate amplification along estimated flat directions can also amplify stochastic noise, limiting improvement despite flat-direction progress being central to final loss reduction.

    Source: §1 · pp. 1–2

  • Method

    MuonM applies curvature-conditioned fast- and slow-decay momentum only in flat directions and uses sphere constraints with parallel transport to prevent norm inflation.

    Source: §1 · p. 2

  • Results

    MuonM consistently achieved lower terminal loss than Muon across dense and MoE models from 0.12B to 2.3B parameters; on dense models, it beat Muon by approximately 0.03.

    Source: §1 · p. 2, §6.2 · p. 9

  • Takeaways & Limitations

    The sphere-constraint technique holds promise as a general building block for developing more efficient optimizers.

    Source: §8 · p. 13

    The optimal learning-rate schedule for sphere-constrained methods remains an open question.

    Source: §8 · p. 13

Abstract

from arXiv · show

Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring severe challenges. Although modern adaptive optimizers such as AdamW and Muon have achieved great success in large-scale pretraining, their reliance on gradient normalization offers limited mitigation of the ill-conditioned curvature. The progress along flat directions (eigen-directions of small eigenvalues), which dominates the final loss reduction, remains relatively slow. To enhance training dynamics along flat directions, we propose a curvature-conditioned multiscale momentum method with sphere constraints, delivering steady acceleration in LLM pretraining. This multiscale momentum, applied only along flat directions, pairs a slow-decay component for noise reduction with a fast-decay component for rapid curvature adaptation, harnessing their complementary strengths. Crucially, we employ a sphere constraint technique to prevent parameter inflation and excessively rapid effective learning rate decay that would otherwise arise from a naive combination. Extensive experiments show that the proposed method significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B--2.3B parameters). Theoretically, we verify the acceleration effect and provide insight into the design principles underlying the flat-direction multiscale momentum.

1 Introduction

The introduction frames slow progress along flat directions as a challenge in noisy, ill-conditioned LLM pretraining. It presents curvature-conditioned multiscale momentum with sphere constraints as a remedy and reports lower terminal loss than Muon across varied training settings.

1 Introduction

  • Noise-dominant gradients and ill-conditioned curvature slow flat-direction progress; AdamW [14] [22] and Muon [20] only partially compensate, while manually amplifying flat-direction learning rates also amplifies noise and can destabilize training.Flat-direction training dynamics primarily drive final loss reduction, but the learning rate is constrained by sharp directions and noise [3] [5].Source: §1 · pp. 1–2
  • The method combines fast- and slow-decay momentum in flat directions, switches it off in sharp directions, and uses sphere constraints with parallel transport to prevent norm inflation and effective learning-rate collapse.Parallel transport also preserves momentum inheritance across changing tangent spaces.Source: §1 · p. 2
  • Across dense and MoE architectures, model sizes from 0.12B to 2.3B, and cosine decay and WSD schedules, MuonM consistently outperforms Muon with lower terminal loss; gains persist under extended token budgets.MuonM denotes the proposed method, formed by appending the suffix “M” to the base optimizer.Source: §1 · p. 2
  • In a linear-regression proxy for LLM pretraining dynamics [16] [19] [25], slow-decay momentum injection along flat directions accelerates optimization by forgetting accumulated noise faster and improving the signal learning rate, yielding lower terminal loss.The analysis supports the design intuition behind flat-direction multiscale momentum.Source: §1 · p. 2

2 Preliminaries and Related Works

The section motivates accelerating progress in flat directions, which dominates final loss reduction, and situates the work among research on adaptive preconditioning, momentum, and effective learning-rate control.

Landscape geometry and training dynamics in deep learning

  • The ill-conditioned loss landscape has sharp directions with large positive Hessian eigenvalues and flat directions with near-zero or negative eigenvalues, where slow progress ultimately dominates final loss reduction.Training oscillates without divergence along sharp directions but makes slow, persistent progress along flat directions; the work focuses on accelerating the latter.Source: §2 · p. 2

Efficient adaptive optimizer design for LLM pretraining

  • AdamW [22] uses coordinate-wise preconditioning, while Muon [20] applies matrix-level preconditioning; this work complements structured preconditioner research by focusing on momentum design.Recent methods [18] [28] develop structured matrix preconditioners to better approximate curvature-related information.Source: §2 · pp. 2–3

Improved momentum mechanism for adaptive optimizers

  • Prior multiscale momentum methods include AggMo [23], AdEMAMix [26], GPA [9], SODA [29], and EMA-Nesterov [49]; this work targets variance reduction under stochastic noise and curvature-dependent time-scale behavior.The work also provides a unified high-order ODE formulation for understanding these algorithmic forms in Appendix D.Source: §2 · p. 3

Controlling the effective learning rate: from weight decay to sphere constraints

  • For normalization-layer parameters, radial evolution affects the effective learning rate, and weight decay or direct norm constraints can control weight-norm growth.The angular update rate ∥∆W∥F /∥W∥F has been treated as the effective learning rate [15] [32] [43] and shown to shape loss trajectories [17] [21] [45]; weight decay controls noise-induced norm inflation, while direct norm constraints [2] [12] [43] [46] offer finer control.Source: §2 · p. 3

3 Illustrative Examples Motivating Flat-Direction Multiscale Momentum

The examples show why slower momentum can reduce noise in flat directions but destabilize sharp ones, motivating a multiscale method restricted to flat directions.

3.1 Momentum Accelerates Flat-Direction Optimization Through Variance Reduction

  • With a fixed true gradient, heavy-ball momentum retains the gradient mean while slower decay reduces estimator variance under Gaussian gradient noise.The setup assumes gk ∼ N(µ, Σ²); smaller α corresponds to slower momentum decay.Source: §3.1 · p. 3
  • Slow momentum is safe only in flat directions, where gradients change slowly; in sharp directions it can introduce bias and instability, while a larger flat-direction learning rate amplifies noise.Figure 1 illustrates faster, less noisy progress with slower decay in the flat direction, but amplified oscillations with slow decay in the sharp direction and with increased flat-direction step size.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 Flat-Direction Multiscale Momentum: Robust Variance Reduction against Projection Errors

  • Flat-direction multiscale momentum interpolates slow momentum for variance reduction with fast momentum for adaptation; excluding slow momentum from sharp directions limits bias while reducing oscillations.For a suitable interpolation coefficient c, the method suppresses noise while keeping bias injection negligible; applying c > 0 also reduces the effective learning rate in the sharp subspace.Source: §3.2 · pp. 4–5
  • With projection errors and varying curvature, flat-direction multiscale momentum yields less noisy acceleration without amplifying sharp-direction oscillations; total-space multiscale momentum is unstable.Figure 2 contrasts stable flat-direction acceleration with instability from applying multiscale momentum across the total space.Source: §3.2 · p. 4, Figure 2 · p. 5

4 Obstacles with Naive Multiscale Momentum: Parameter Inflation and Effective Learning Rate Collapse

Naive flat-direction multiscale momentum gives only a diminishing loss gain because it inflates parameter norms and accelerates effective learning-rate decay; increasing weight decay does not solve the problem.

4 Obstacles with Naive Multiscale Momentum: Parameter Inflation and Effective Learning Rate Collapse

  • The method treats the bottom 90% singular spaces of each momentum matrix as flat directions [56], adding slow momentum there alongside Muon-preconditioned fast momentum.It uses row-wise normalization for slow momentum to match the m_sign term’s norm, inspired by near-zero inter-row Hessian observed in [52].Source: §4 · pp. 5–6
  • Less than 0.01 terminal loss improvement remains after the gain from adding flat-direction slow momentum progressively diminishes, alongside parameter-norm growth and faster effective learning-rate decay.The 0.12B dense-model comparison thoroughly tuned both Muon and χ and α_slow; Figure 3 shows the validation loss and an up-projection block’s Frobenius norm evolution.Source: Figure 3 · p. 6, §4 · p. 6
  • A sharp early loss drop is followed by Muon gradually catching up, although flat-direction slow momentum retains a marginal edge.The passage attributes this evolution to rapid effective learning-rate decay in the early phase.Source: §4 · p. 6
  • Raising weight decay to λ = 0.2 restores the parameter norm to the baseline level but makes loss higher, so hyperparameter tuning alone is insufficient.The default weight decay coefficient is λ = 0.1.Source: §4 · p. 6

5 The Flat-Direction Multiscale Momentum Method with Sphere Constraints

The method replaces weight decay with Frobenius sphere constraints and applies multiscale momentum to flat directions through manifold-aware updates. Learnable-radius parameterizations decouple radial and angular updates.

5 The Flat-Direction Multiscale Momentum Method with Sphere Constraints

  • The method replaces weight decay with sphere constraints to address weight norm inflation.It restricts weights to fixed-norm Frobenius spheres.Source: §5 · p. 6, §5.1 · p. 6

5.1 Preliminaries on Manifold Optimization

  • Manifold updates project directions onto the sphere’s tangent space, retract iterates back to the sphere, and parallel-transport momentum between tangent spaces.Parallel transport preserves vector norms and helps prevent distortion of accumulated momentum as the tangent space changes.Source: §5.1 · pp. 6–7

5.2 The Proposed Approach

  • Transformer matrices use a learnable scalar radius with fixed-norm direction; embedding and output matrices instead use row-wise learnable scales.Vector parameters are re-parameterized analogously, and the base optimizer on this parameterization is denoted with the suffix S, as in MuonS.Source: §5.2 · p. 7
  • MuonM applies multiscale momentum under sphere constraints, rescaling fast- and slow-momentum updates to radius R so the learning rate acts as angular velocity.Its flat-direction projection Qk is estimated by online power iterations over top singular subspaces of the fast momentum.Source: §5.2 · pp. 7–8
  • The method adds marginal computation, memory, and communication overhead relative to Muon.The extra memory is roughly one additional set of model parameters, while manifold operations avoid dense matrix–matrix products and the flat-direction projection uses lightweight msign operations on smaller matrices.Source: §5.2 · p. 8

6 Experiments

MuonM consistently achieves the lowest terminal loss across dense and MoE models, while ablations show that flat-direction projection and slow-momentum parallel transport are important to its performance.

6 Experiments

  • Experiments use a 350B-token pretraining corpus, with budgets of approximately 100 tokens per parameter for dense models and 500 tokens per activated parameter for MoE models.The passage says these budgets exceed the 20 TPP Chinchilla-optimal data reference and align more closely with industrial pretraining practice.Source: §6 · p. 8

6.1 Validation of Critical Algorithm Components

  • Replacing flat-direction projection with sharp-direction projection raises loss; slow-momentum parallel transport is indispensable, whereas removing it for fast momentum has negligible effect.The proposed explanation is that slow momentum spans more updates as the tangent space changes, while fast momentum decays quickly; their example half-lives are approximately 700 and 14 steps, respectively.Source: Figure 5 · p. 8, §6.1 · pp. 8–9

6.2 Results on Dense Models

  • All sphere-constrained methods outperform weight-decayed Muon, with MuonS achieving the lowest loss among them; MuonM beats AdEMAMix and EMA-Nesterov, with gains of -0.019 over MuonS and -0.028 over Muon versus -0.008 from adding flat-direction momentum alone.The authors attribute the sphere-strategy advantage to adaptive radii across heterogeneous blocks.Source: Figure 6 · p. 9, §6.2 · p. 9
  • Across dense models from 0.12B to 1.4B parameters, MuonM consistently achieves the lowest terminal loss, outperforming MuonS by approximately 0.02 and Muon by approximately 0.03.The comparison is against well-tuned MuonS and Muon baselines.Source: §6.2 · p. 9, Figure 7 · p. 10
  • Across 100–1000 TPP on the 0.12B dense model, MuonM has the lowest terminal loss at every budget, with a widening performance gap as training increases.The searched optimal learning rate for MuonS and MuonM decays from 1e-2 at 100 TPP to 7e-3 at 1000 TPP, versus Muon's decline from 5e-3 to 1e-3.Source: §6.2 · pp. 9–10, Figure 8 · p. 10

6.3 Results on MoE Models

  • On 0.64B (A0.13B) and 2.3B (A0.36B) MoE models, MuonM reduces terminal loss versus Muon and MuonS, and also achieves the lowest loss under WSD on the 0.64B model.The results support the extension of MuonM's advantages from dense architectures to MoE models.Source: §6.3 · p. 10

7 Theoretical Analysis

The analysis uses a feature-space linear-regression proxy to study flat-direction multiscale momentum. Its theoretical results characterize excess-risk components and show that slow momentum accelerates convergence under the stated conditions.

7.1 Problem Formulation

  • The analysis uses feature-space linear regression as a quadratic proxy for pretraining, with Gaussian features of diagonal covariance and power-law spectrum and target assumptions.The power-law spectrum exponent characterizes feature-model capacity, while the task parameter measures relative task difficulty.Source: §7 · p. 10, §7.1 · p. 11

7.2 Theoretical Results

  • Theorem 7.2 bounds excess risk through signal-learning error and noise accumulation for afull Tk ≲ d^ν; this covers typical k ≲ d, while longer-time matching bounds remain unavailable.The theorem assumes the power-law model and additional stated conditions.Source: §7.2 · p. 12
  • For the same αfast and γ with learning rate tuned, incorporating slow momentum (χ > 0) accelerates convergence; larger χ with proportionally smaller η reduces both error components.The constant-learning-rate corollary states its result under r ≥ 4 and the conditions of Theorem 7.2.Source: §7.2 · p. 12
  • Slow momentum yields proxy-optimal loss-reduction and iteration-complexity speedups, which become more pronounced as χ/αslow increases.Corollary 7.5 characterizes these speedups using the proxy-optimal learning rate.Source: §7.2 · pp. 12–13

8 Conclusion and Discussion … B.1 Experimental Details of Section 4

MuonM combines flat-direction multiscale momentum with sphere constraints to accelerate LLM pretraining, and the conclusion reports consistent acceleration of Muon with substantial loss reductions across diverse architectures and training setups.

8 Conclusion and Discussion

  • MuonM consistently accelerates Muon with substantial loss reductions across diverse architectures and training setups; sphere constraints resolve effective learning-rate collapse and may support more efficient optimizers.The conclusion presents sphere constraints as a more effective foundation for flat-direction multiscale momentum than standard weight decay.Source: §8 · p. 13
  • The study leaves the optimal learning-rate schedule for sphere-constrained methods open and identifies slow-momentum preconditioner design as a possible route to further acceleration.Source: §8 · p. 13

A.1 Estimating Flat Directions

  • A.1 motivates flat-direction projection with empirical alignment between top eigenspaces of G^⊤G and GG^⊤ and the corresponding row and column Hessians [56], then tracks sharp subspaces using online power iteration and msign.The method uses the Nesterov-type momentum as a stabilized gradient proxy and replaces computationally expensive QR decomposition with msign; the example sharp-subspace ratio is d = 0.1n.Source: §A.1 · p. 18, Figure 10 · p. 18
  • Using d = 0.1n for projection matrices reduces Newton-Schulz computation to approximately 1% of the full-dimensional cost, with additional memory mainly for a right projection matrix smaller than the momentum.The stated cost ratio is approximately (d/n)^2.Source: §A.1 · p. 18

A.2 Update Rules of MuonM for Different Parameter Types

  • MuonM applies Muon to matrix parameters, Adam with sphere constraints and slow momentum to partial vector blocks, and Adam directly to scalar parameters.The parameter categories include Transformer weight matrices; biases, RMSNorm vectors, embedding/output rows, and learnable grouped radii; and native architectural scalars.Source: §A.2 · p. 19
  • Embedding and output matrices use row-wise sphere constraints, while their flat-direction projection selects rows for acceleration; A.1 also selects embedding/output rows by second-moment norm, motivated by token-wise Hessian block structure [52].The selected-row count is d ≪ V, with d = 0.1V given as an example.Source: §A.1 · p. 18, §A.2 · p. 19

B Experimental Details

  • The experimental implementation uses six Newton-Schulz iterations with Polar Express coefficients [1] for msign, which the passage says is more precise than the method in, and searches peak learning rates from 5e-4 to 2e-2.Source: B Experimental Details · p. 19

B.1 Experimental Details of Section 4

  • The best Section 4 configuration uses Muon learning rate 0.005, αslow = 0.002, and χ = 0.1, with αfast = 0.05 and a 10% sharp-subspace ratio.The search varied αslow over {0.005, 0.002, 0.001, 0.0005} and χ over {0.1, 0.2, 0.5}.Source: §B.1 · p. 19

B.2 Experimental Details of Section 6.2 · C Additional Experimental Results

The experimental details specify optimizer configurations, Muon and MuonM settings, dense and MoE model architectures, and learning-rate schedules used in the evaluations.

B.2 Experimental Details of Section 6.2

  • For the 0.12B dense-model optimizer comparison, all optimizers use cosine decay, with learning rates and other hyperparameters selected by grid search.MuonH and SSO use learning rates 1 × 10−2 and 5 × 10−3, respectively; SSO uses radius 2.0. AdEMAMix selects (lr, β3, α) = (3 × 10−3, 1 × 10−3, 5.0), while EMA-Nesterov selects (lr, βmax, γ) = (3 × 10−3, 0.9, 0.99).Source: §B.2 · p. 20
  • Muon updates 2D transformer weight matrices while AdamW handles RMSNorm, embeddings, and the output layer.Muon uses Nesterov momentum with decay 0.95, weight decay 0.1, gradient clipping threshold 1.0, and learning-rate scaling by 0.2 max{m, n} for an m × n matrix.Source: §B.2 · p. 20
  • MuonM uses χ = 0.2, αfast = 0.05, αslow = 0.001, and treats each block’s top 10% subspace as sharp directions.A warmup adjusts the slow momentum coefficient from αfast toward αslow during the learning-rate warmup to accommodate rapid early gradient variation.Source: §B.2 · p. 20
  • Experiments cover dense decoder-only Transformers and MoE models, with architecture configurations summarized in Tables 1 and 2.Dense models use RoPE, SwiGLU, RMSNorm, Pre-Norm, and MHA; MoE models use MLA, one shared expert and partially activated experts, with a dense first layer and auxiliary-loss-free load balancing. Table 1 models use sequence length 1024; Table 2 models use sequence length 4096 and batch size 1024.Source: §B.2 · p. 20, Table 1 · p. 20, Table 2 · p. 21
  • The evaluated schedules use min{1000, T/50} warmup steps and decay to lr_min = 0.05 × lr_max.Cosine scheduling follows linear warmup with cosine decay; warmup-stable-decay holds lr_max through 70% of steps, then uses minus-sqrt decay over the final 30%.Source: §B.2 · pp. 20–21
  • On the 0.64B MoE model, searched peak learning rates are 5e-3 for Muon and 7e-3 for MuonS, while MuonM is set to 4.5e-3.MuonM’s rate is approximately 2/3 of MuonS’s, without a separate search.Source: §B.2 · p. 21

C.1 Results with WSD Learning Rate Schedules

Under the WSD schedule, MuonM achieves the lowest terminal loss among Muon, MuonS, and MuonM on the 0.6B MoE model.

C.1 Results with WSD Learning Rate Schedules

  • MuonM achieves the lowest terminal loss under WSD among Muon, MuonS, and MuonM.Figure 11 shows their validation-loss trajectories on the 0.6B MoE model.Source: §C.1 · p. 21, Figure 11 · p. 21
  • Peak learning rates were 0.005 for Muon, 0.007 for MuonS, and 0.0045 for MuonM.Source: §C.1 · p. 21

D A Continuous-Time Perspective for Unifying Enhanced Momentum Methods … E.1 Variance Reduction of Multiscale Momentum

The paper relates several enhanced momentum methods through continuous-time ODEs and shows that increasing the slow-momentum weight monotonically reduces stationary variance in an illustrative Gaussian analysis. Its ODE framework extends from two-timescale momentum to n timescales.

D A Continuous-Time Perspective for Unifying Enhanced Momentum Methods

  • AdEMAMix, GPA, EMA-Nesterov, SODA, and the proposed method share a continuous-time ODE form when preconditioning is ignored.The derivation assumes gradient-noise-free conditions and constant hyperparameters; GPA and EMA-Nesterov use Heavy Ball as their base optimizer.Source: D A Continuous-Time Perspective for… · p. 21
  • The proposed MuonM example yields a continuous-time counterpart and a third-order ODE after omitting the sphere constraint and Muon preconditioning.Constants are absorbed into step sizes for notational simplicity in the discrete-time recursion.Source: D A Continuous-Time Perspective for… · p. 22
  • GPA maintains base, exponentially averaged, and gradient-evaluation interpolation sequences; its preconditioned Heavy Ball instantiation has a continuous-time limit [9].The generic GPA framework permits an arbitrary base optimizer.Source: D A Continuous-Time Perspective for… · p. 22
  • AdEMAMix augments AdamW with a substantially slower first-moment EMA, whose continuous-state equations are derived after removing preconditioning, bias correction, and weight decay [26].Eliminating the two momentum states yields a corresponding parameter equation.Source: D A Continuous-Time Perspective for… · p. 23
  • EMA-Nesterov replaces the one-step Nesterov look-ahead direction with an EMA of parameter increments and has a continuous-time limit without preconditioning or weight decay [49].The wrapper can be expressed using a base-optimizer map and an EMA look-ahead state.Source: D A Continuous-Time Perspective for… · p. 23
  • SODA combines gradient averaging, dual optimism, primal averaging, and primal extrapolation; optimism and extrapolation together yield an implicit preconditioner [29].The section derives a continuous limit and a closed equation for the extrapolated iterate.Source: D A Continuous-Time Perspective for… · p. 24

E.1 Variance Reduction of Multiscale Momentum

  • Increasing the slow-momentum weight monotonically reduces stationary variance when 0 < αslow < αfast ≤ 1 and c ∈ [0, 1].The analysis uses an illustrative Gaussian example and considers update momentum whose stationary mean is preserved.Source: §E.1 · p. 25
  • Slow momentum reduces stationary variance by a factor of up to αslow(2 − αfast) relative to fast momentum alone.The limiting regimes compare fast momentum alone at c = 0 with slow momentum alone at c = 1.Source: §E.1 · pp. 25–26

E.2 Weight Norm Inflation by Slow Momentum

The section analyzes how slow momentum inflates weight norms and identifies the conditions under which stationary mean-square parameter norms remain finite.

E.2 Weight Norm Inflation by Slow Momentum

  • The analysis assumes a scale-invariant loss, i.i.d. Gaussian stochastic gradients, and a fixed flat-direction projection of rank q ≈ d.The motivating intuition is that Brownian-motion-like movement drives norm growth while weight decay suppresses it.Source: §E.2 · p. 26
  • Slow momentum inflates the weight norm through the positive coefficient χ and the large-n dominance Aslow,n ≫ Afast,n.The proposition assumes both processes start from the same deterministic w0 and ρ := 1 − ηλ > 0.Source: §E.2 · p. 26
  • Stationary mean-square parameter norms are finite if and only if λ > 0.Vχ is finite if and only if ρ < 1, equivalent to λ > 0.Source: §E.2 · pp. 26–27

F Proof · F.1 Useful Lemmas

This section assembles technical lemmas for the proofs, covering links between late-training gradient noise and Hessian curvature, parameter–momentum recursions and kernel bounds, convolution and approximation-error estimates, eigenvalue localization, and resolvent perturbations and Riesz spectral projections.

F.1 Useful Lemmas

  • In the later phase, when E ≲ σ^2, gradient noise aligns well with the Hessian and the stochastic gradient is noise-dominated; early noise is dominated by the loss scale.Source: F Proof · p. 27
  • Lemma F.4 gives a discrete parameter–momentum recursion equivalent to algorithm (8), with shared noise in all three entries.Source: F Proof · pp. 27–28
  • For sufficiently small positive learning rate, the root issuing from 1 remains separated from the two roots issuing from ρ_fast and ρ_slow, and the largest-root displacement is bounded by 2ax ≤ α_slow/4.The displacement tends to zero as x tends to zero.Source: F Proof · pp. 28–29
  • Lemma F.6 defines complementary oblique projections: P0 is associated with the largest eigenvalue, while P1 projects onto the complementary two-dimensional subspace.At x = 0, P0(0) = diag(1, 0, 0), and P0 is differentiable there.Source: F Proof · p. 30
  • Under Assumption F.1 and d ≥ 2(r + 1), Lemma F.7 establishes uniform kernel bounds, including lower and upper estimates, forgetting-kernel and signal-rate bounds, and a conditional bound for r ≥ 4.Source: F Proof · pp. 32, 34, 35

F Proof

  • The proof applies Lemma F.2 to prove (96).Source: F Proof · p. 36
  • The approximation-error analysis separates the effective gain aq exactly from a discrete boundary difference, without spectral approximation.It also establishes εep(k) ≤ 1/16.Source: F Proof · pp. 41, 44
  • For sufficiently large d and τ0 and sufficiently small η0, Lemma F.12 ensures admissibility for the decreasing schedule considered.The schedule is ηi = η0(1 + i/τ0)^−γ.Source: F Proof · p. 45
  • Under the stated admissible hyperparameter conditions, the proof establishes a uniform upper bound and asymptotic bounds for Ek.The proof organizes these as separate steps before concluding Theorem 7.2.Source: F Proof · pp. 45–47
  • The proof of Corollary 7.4 combines estimates of the noise accumulation and signal terms to obtain (12).The signal estimate uses the lower inequality in (11).Source: F Proof · pp. 47–48

F.1 Useful Lemmas

  • The Gershgorin disk theorem places every eigenvalue of M in the union of its row disks, and an isolated connected component of k disks contains exactly k eigenvalues counted with algebraic multiplicity.Source: §F.1 · p. 50
  • When ∥RA(z)E∥<1, the perturbed resolvent satisfies RA+E(z) = (I − RA(z)E)−1RA(z), with ∥RA+E(z)∥ ≤ ∥RA(z)∥/(1 − ∥RA(z)E∥).The result applies when zI − A is invertible.Source: §F.1 · p. 50
  • For a contour Γ with no eigenvalue of A on it, PΓ(A) is an idempotent projection onto the generalized eigenspaces inside Γ and commutes with A.Source: §F.1 · p. 50

Related papers

Loading 2608.28442v1…