PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?

Kaifei Wang, Zikang Shan, Liwei Wang, Yinyu Ye, Han Zhong

arXiv:2610.20413v1cs.LGcs.AImath.OCstat.ML

TL;DR

The paper asks when first-moment momentum benefits RLVR, where optimizer choices have received limited attention and empirical results are inconsistent. It analyzes bias-corrected EMA in a response-level bandit and finds that coefficient scaling can preserve gains amid sampling noise; LLM experiments support larger coefficients.

  • Problem

    Optimizer design in RLVR has received relatively limited attention, and prior experiments report that momentum does not consistently improve performance.

    Source: §1 · p. 1

  • Method

    The paper analyzes bias-corrected EMA in a KL-regularized response-level bandit, studying exact-gradient optimization and sampled-response updates.

    Source: §3.1 · p. 3, §5 · p. 5, §7 · p. 10

  • Results

    Increasing β within the safe regime strictly accelerates exact-gradient optimization; on Qwen2.5-32B, β1 = 1 −10−9 reaches 46.1% mean accuracy versus 44.3% at β1 = 0.9.

    Source: §6.3 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    The findings suggest using first-moment coefficients substantially above the conventional β = 0.9 to better exploit the optimizer’s momentum information.

    Source: §7 · p. 10

    The model assumes deterministic binary rewards and a suboptimal initial policy with both correct and incorrect responses available.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Momentum is a classical tool for accelerating convex optimization and a standard component of modern large language model training. Yet recent empirical work has questioned the usefulness of first-moment momentum in reinforcement learning with verifiable rewards (RLVR). We study this question through a bandit formulation of RLVR with bias-corrected exponential moving average (EMA) momentum, where the EMA coefficient $\beta$ controls how quickly the weights on past gradients decay. When policy gradients are evaluated exactly, we prove that increasing $\beta$ within a range that prevents overshoot strictly accelerates optimization. When gradients are estimated from sampled responses, a positive expected gain can be obscured by sampling fluctuations. We show that as the learning rate $\eta$ decreases, the expected gain with fixed $\beta$ vanishes faster than these fluctuations, whereas increasing $\beta$ to keep $\eta/(1-\beta)$ within the safe regime preserves a nonvanishing gain that eventually dominates them. Bandit simulations quantitatively validate these predictions. RLVR experiments on Qwen3-4B-Base using SGD with EMA and AdamW support our principle of choosing larger coefficients in practical LLM post-training. On Qwen2.5-32B with AdamW, simply increasing $\beta_1$ improves mean validation accuracy by nearly $2$ percentage points over the conventional choice, demonstrating the principle's effectiveness at a larger model scale.

1 Introduction

The paper investigates when first-moment EMA momentum helps RLVR, where optimizer design has received limited attention and prior experiments report inconsistent benefits. It analyzes exact and sampled gradients and tests coefficient choices in bandit simulations and LLM post-training.

1 Introduction

  • Optimizer configuration in RLVR has received relatively limited attention, despite its use of Adam or AdamW with conventional first-moment settings.RLVR samples responses from the current policy and evaluates them with verifiable rewards, so its training-data distribution evolves with the policy.Source: §1 · p. 1
  • The analysis uses a KL-regularized response-level bandit formulation and examines natural policy gradient with softmax policies and bias-corrected EMA.The framework is chosen to study policy improvement while controlling changes in the policy distribution.Source: §1 · p. 1
  • With exact gradients, increasing EMA’s coefficient within a safe regime strictly accelerates optimization, while sampled-response fluctuations can obscure the expected gain.For fixed β, the expected gain vanishes faster than sampling fluctuations as the learning rate decreases; scaling β with the learning rate preserves a nonvanishing gain that eventually dominates them.Source: §1 · p. 2
  • Bandit simulations validate the predicted gain and fluctuation scales, while Qwen3-4B-Base GRPO experiments support larger coefficients with SGD with EMA and AdamW.Pairing larger coefficients with smaller learning rates yields more stable training with comparable or slightly improved performance.Source: §1 · p. 2
  • On Qwen2.5-32B, simply increasing AdamW’s first-moment coefficient improves mean validation accuracy by nearly 2 percentage points over the conventional choice.Source: §1 · p. 2

2 Related work

Prior work studies momentum across RLVR, conventional reinforcement learning, and supervised learning or stochastic optimization. This paper focuses on ordinary EMA momentum in RLVR, where responses sampled from an evolving policy receive reward feedback.

Optimization and momentum in RLVR

  • In RLVR, Mukherjee et al. (2026) report that SGD can match or outperform AdamW and that momentum has no consistent benefit, motivating study of its dependence on coefficient, learning rate, and sampling noise.Other RLVR optimizer work examines momentum-based normalization, spectral transformations, parameter-space exploration, and accumulated AdamW momentum after clipping.Source: §2 · p. 2

Momentum in reinforcement learning

  • Conventional RL research studies momentum for acceleration and variance reduction, mainly analyzing convergence or sample complexity and evaluating control and Atari benchmarks.This paper instead studies bias-corrected averaging of past natural gradients in a KL-regularized RLVR bandit and examines GRPO experiments with language models.Source: §2 · p. 2

Momentum in supervised learning and stochastic optimization

  • Supervised-learning and stochastic-optimization studies analyze momentum under learning-rate scaling, while language-model pretraining methods use longer gradient memory or scheduled momentum on next-token prediction.This paper addresses RLVR, where reward feedback comes from responses sampled from the evolving policy.Source: §2 · p. 2

3 Preliminaries

The preliminaries specify a response-level RLVR model and the optimization framework used for the analysis. They define the reward and reference-policy objective, then introduce softmax policies, natural policy gradient, and bias-corrected EMA updates.

3.1 Model and parameterization

  • The model samples a response for each prompt, scores it with a verifier, and maximizes expected reward while penalizing divergence from a fixed reference policy.Rewards are deterministic and binary; the reference policy has full support, and the initial policy is assumed suboptimal with both correct and incorrect responses available.Source: §3.1 · p. 3
  • A response-level softmax over logits parameterizes the conditional distribution of responses to a prompt.The paper presents this as a tractable abstraction of token-level softmax policies in autoregressive LLMs.Source: §3.1 · p. 3

3.2 Training dynamics

  • Natural policy gradient preconditions the policy gradient with the Fisher information matrix; sampled updates replace the gradient with an unbiased estimate while computing the KL gradient and Fisher matrix exactly.The analysis connects NPG’s policy-change control to TRPO, PPO, and GRPO.Source: §3.2 · p. 3
  • EMA weights current and past natural gradients across updates, and bias correction normalizes the weights to sum to one; β = 0 recovers the update without EMA.The coefficient β controls the contribution of the current gradient, while the stochastic update uses the same gradient weights.Source: §3.2 · p. 4

4 Deterministic case: acceleration with exact gradients

The deterministic analysis compares exact natural-policy-gradient iterates with and without bias-corrected EMA from the same initialization. It shows that increasing the coefficient within a no-overshoot range strictly improves the objective after the specified updates.

4 Deterministic case: acceleration with exact gradients

  • The comparison uses the same initial parameters, learning rate, and update count for EMA and vanilla NPG.Both runs start with zero momentum.Source: §4 · p. 4
  • Before overshoot, exact-gradient iterates approach the optimum along a common direction, so EMA averages the current natural gradient with earlier, larger gradients.The parameter error remains a scalar multiple of the initial error while the relevant iterates stay positive.Source: §4 · p. 5
  • Under 0 < ηγ < 1 and T ≥ 2, Theorem 4.2 considers β1 < β2 in [0, ((1 − ηγ)/(1 + ηγ))²].Source: §4 · p. 5
  • The deterministic result motivates examining why sampled training may fail to realize a reliable improvement and how to recover the gain.Within the safe range, EMA’s deterministic acceleration comes from temporally reallocating natural-gradient contributions.Source: §4 · p. 5

5 Stochastic case: sampling fluctuations and expected gain

With sampled responses, EMA’s expected gain can be obscured by training fluctuations: a fixed coefficient’s gain vanishes with the learning rate, while scaling the coefficient can preserve an observable gain. Increasing the coefficient can also reduce run-to-run variance while maintaining comparable expected performance under the same update and sample budgets.

5 Stochastic case: sampling fluctuations and expected gain

  • O(η/B) variance in the objective difference persists regardless of β, so fluctuations make EMA difficult to distinguish when its expected gain is small relative to √(η/B).The expected difference is characterized by the exact objective difference.Source: §5 · p. 6

5.1 A fixed EMA coefficient gives a vanishing expected gain

  • With fixed β, the expected gain is positive but only order η, so sampling fluctuations can overwhelm it and make individual runs difficult to distinguish.This offers a possible explanation for the limited benefits reported in prior RLVR experiments.Source: §5.1 · p. 7

5.2 Scaling the EMA coefficient with the learning rate

  • Holding ρ = η/(1−β) at a positive safe value preserves a nonvanishing expected gain that eventually dominates sampling fluctuations as η decreases.The safe regime requires ρ ≤ 1/(4γ).Source: §5.2 · p. 7

5.3 Reducing variability across runs with EMA

  • At the same update count and sample budget, increasing β can offset a smaller learning rate; sampled runs then have an O(η/B) expected-objective difference and an asymptotic variance ratio below one.The parameters obtained with exact gradients match after the same number of updates.Source: §5.3 · p. 8

6 Experiments

Bandit simulations validate that scaling momentum as the learning rate decreases preserves gains against sampling fluctuations, while RLVR experiments support larger momentum coefficients in practical post-training. On Qwen2.5-32B, increasing AdamW’s first-moment coefficient improves mean validation accuracy by nearly 2 percentage points over the conventional setting.

6.1 Validating theoretical predictions in bandits

  • At η = 1/4096, scaled β = 1 −4η yields SNRs of 3.7647 and 3.3982, versus 0.0453 and 0.0406 with fixed β = 0.9 under uniform and random initialization.The standard deviation scales as √η, while the fixed-β mean gain scales as η and the scaled-β gain approaches a positive constant; EMA’s probability of beating an independent no-EMA run approaches 1/2 with fixed β and 1 with scaled β.Source: §6.1 · p. 8

6.2 Practical LLM post-training

  • On Qwen3-4B-Base, tuned larger coefficients substantially improve validation performance over both conventional 0.9 momentum and no first-moment momentum for SGD with EMA and AdamW.The compared settings include β = 0.9992 for SGD and β1 = 1 −10−6 for AdamW.Source: §6.2 · pp. 8–9, Figure 2 · p. 9
  • With the same update and sample budgets, pairing larger momentum with a smaller learning rate preserves low cross-seed variability and gives slightly better performance than larger-rate runs with conventional momentum.The comparisons are shown for SGD with EMA and AdamW.Source: §6.2 · p. 9, Figure 2 · p. 9

6.3 Extending to a larger model

  • 46.1% mean accuracy with β1 = 1 −10−9 at step 150 exceeds 44.3% with β1 = 0.9 on Qwen2.5-32B.The conventional setting is similar to the 44.2% no-first-moment baseline.Source: §6.3 · p. 9, Figure 3 · p. 9

7 Conclusion

The paper concludes that sampling noise can obscure momentum’s benefits and that increasing EMA within a safe regime can recover them. Its conclusion points to larger-than-conventional coefficients, while the included appendix passage describes how sampled-gradient calculations are obtained.

7 Conclusion

  • EMA strictly improves the objective with exact gradients within a safe range, while suitably increasing β with sampled gradients yields consistent improvement despite sampling noise.The conclusion suggests coefficients substantially above conventional β = 0.9 to better exploit first-moment information.Source: §7 · p. 10

AI use statement

  • The authors report AI assistance with derivations, literature searches, proof checking, and language editing, and say they reviewed the material and take responsibility.Source: AI use statement · p. 10

A Proof of Proposition 3.1

  • The sampled-gradient derivation evaluates the KL term exactly and premultiplies the exact and sampled expressions by their corresponding Fisher pseudoinverses.The passage says this establishes the stated natural-gradient identities.Source: A Proof of Proposition 3.1 · p. 13

B Proofs of Lemma 4.1 and Theorem 4.2

The proof establishes that, in the stated safe range, increasing the EMA coefficient reduces the residual from the optimum and strictly improves the objective. It derives a scalar recurrence for the parameter trajectory and uses residual comparison to obtain the objective result.

B Proofs of Lemma 4.1 and Theorem 4.2

  • The parameter identity follows by induction from the EMA update and the defining recurrence for f_t; the proof also uses the unique optimal policy to establish the objective comparison.The optimal policy is characterized by equality when π_θ = π_θ⋆.Source: B Proofs of Lemma 4.1 and Theorem 4.2 · pp. 13–14
  • The proof establishes positivity of f_t in the safe range and shows that its ratios decrease with β, yielding a smaller residual for larger coefficients.It then converts this residual comparison into strict objective improvement.Source: B Proofs of Lemma 4.1 and Theorem 4.2 · p. 14
  • For 0 < β1 < β2, the proof obtains 0 < f_T(β2) < f_T(β1) < (1 −ηγ)^T = f_T(0) for T ≥ 2.Source: B Proofs of Lemma 4.1 and Theorem 4.2 · p. 15

C Proof of Lemma 5.1

The appendix controls sampled-update noise by comparing an auxiliary bounded-noise process with the sampled process, then expands the objective to characterize its expectation and variance. It also supplies uniform bounds on residual and noise-response quantities used in these arguments.

C Proof of Lemma 5.1

  • Noise-response coefficients track how each update’s sampling noise affects later parameters; the covariance coefficient is a deterministic positive-semidefinite weighted sum of noise covariances along the exact trajectory.The proof states that the response coefficients depend on the update parameters, not on realized samples.Source: C Proof of Lemma 5.1 · p. 15

C.1 Proof of Lemma C.1

  • Uniform response bounds control f_t and Ψβ for β ∈ [0, 1) and tη ≤ τ, supporting the appendix’s noise and objective estimates.The stated residual bound is |f_t(β)| ≤ e^γτ.Source: C Proof of Lemma 5.1 · p. 16, §C.1 · p. 18

C Proof of Lemma 5.1

  • The proof constructs an auxiliary process that agrees with the sampled process until leaving a unit neighborhood of the exact path, then bounds exit probability and moments.This supports transferring the objective expansion from the auxiliary process to the sampled one.Source: C Proof of Lemma 5.1 · pp. 16–18
  • The objective expansion bounds the quadratic correction by O(η/B) uniformly in β and transfers the expected objective from the auxiliary process to the sampled process.Source: C Proof of Lemma 5.1 · pp. 17–18

D Proof of Proposition 5.2

  • For the variance argument, subtracting the square of the expected-objective expansion cancels the Hessian term, and independence makes the two comparison variances add.The proof notes that under its nondegeneracy condition, the no-EMA term stays bounded away from zero as η → 0.Source: D Proof of Proposition 5.2 · p. 19

E Proof of Theorem 5.3

The proof establishes the small-learning-rate behavior of fixed-β EMA: its expected gain vanishes, and its winning probability tends to one half. It also shows that keeping η/(1−β) away from zero preserves a positive expected gain.

E Proof of Theorem 5.3

  • For fixed β, the exact-path objective gain is approximated using f_T(β), with a strictly positive coefficient independent of η, β, and B under the nondegeneracy condition.The proof compares sampling corrections and bounds the Hessian and covariance terms to obtain the expected gain.Source: E Proof of Theorem 5.3 · pp. 19–20
  • Under Theorem 5.3’s assumptions, the standardized objective difference converges to a standard normal distribution.The proof combines this normal limit with a vanishing mean-to-standard-deviation ratio to derive the limiting winning probability.Source: E Proof of Theorem 5.3 · p. 20, §E.1 · pp. 23–24

F Vanishing expected gain as η/(1 −β) →0

G Proof of Theorem 5.4

The proof represents the EMA path continuously when β=1−η/ρ, then uses that approximation to establish positive gain in the safe regime. The approximation error is controlled as η decreases.

G.1 Proof of Lemma G.1

  • The continuous residual f(s; ρ) approximates the discrete path f_t(β) with O(η) error when β=1−η/ρ and Tη≤τ.The proof constructs a unique continuous solution and bounds the discrete-to-continuous change.Source: G Proof of Theorem 5.4 · p. 25, §G.1 · p. 26

G Proof of Theorem 5.4

  • When β=1−η/ρ and 0<ρ≤1/(4γ), the EMA coefficient lies in the safe range for sufficiently small η.The proof uses residual-ratio bounds to establish positive gain within this range.Source: G Proof of Theorem 5.4 · p. 25
  • For fixed ρ in the safe range, the expected gain tends to C(τ, γ, ρ)ρ > 0 and eventually dominates sampling fluctuations.The proof bounds the variance by O(η/B) and applies Chebyshev’s inequality to the winning probability.Source: G Proof of Theorem 5.4 · p. 26

H Proof of Theorem 5.5

The proof matches endpoints across runs, then compares their expected objectives and variances. Its variance analysis uses a continuous covariance limit and the long-time decay rate of the residual.

H Proof of Theorem 5.5

  • The proof establishes an expected-objective comparison after matching endpoints at finite step size.It selects a smaller learning rate for the second run while matching the residual endpoint.Source: H Proof of Theorem 5.5 · p. 28
  • The matched run’s limiting optimization time is determined by equating its continuous residual with the first run’s residual.The continuous residual is strictly decreasing, giving a unique matching time.Source: H Proof of Theorem 5.5 · p. 28

H.1 Proof of Lemma H.1

  • The continuous covariance coefficient converges as η→0, and the proof computes its long-time limit to establish the variance comparison.The analysis derives the long-time residual decay rate κ(ρ) and uses it to obtain the limiting variance ratio.Source: H Proof of Theorem 5.5 · pp. 29–30

I.1 Bandit results

Across four bandit settings, the observed gain and fluctuation trends match the theoretical predictions as η decreases. The LLM experiments use shared training and evaluation settings across optimizer configurations.

I.1 Bandit results

  • The bandit comparison uses fixed β=0.9 and scaled β=1−4η against β=0, with 50,000 training replicates per configuration.It varies reward vectors and uniform or random policy initialization; Figure 4 covers the 5:5 reward-arm settings.Source: §I.1 · pp. 32–33, Figure 4 · p. 33
  • Across all four settings, standard deviation scales as √η, while mean gain scales as η for fixed β and approaches a positive constant for scaled β.The winning probability approaches 1/2 with fixed β and 1 with scaled β; the results span different reward vectors and policy initializations.Source: §I.1 · p. 33

I.2 Large language model experiments

  • The LLM experiments use Qwen3-4B-Base and Qwen2.5-32B, with 1,000 DAPO-17k problems held out for validation.Validation reports mean per-problem accuracy over eight sampled responses per problem.Source: §I.2 · pp. 33–34, Table 2 · p. 34

Related papers

Loading 2610.20413v1…