PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization

Yangyi Fang, Jiaye Lin, Xiaoliang Fu, Cong Qin, Haolin Shi, Chaowen Hu, Lu Pan, Ke Zeng, Xunliang Cai

arXiv:2602.19208v2cs.LGcs.AI

TL;DR

RLVR must allocate rollouts across problems with heterogeneous informativeness while controlling softmax-induced gradient attenuation and unstable updates. DynaMO addresses these issues with variance-minimizing sequence-level allocation and gradient-aware token-level advantage modulation. Across benchmarks and scales, it reports consistent improvements over strong baselines, though evaluation is limited to the Qwen family.

  • Problem

    RLVR methods uniformly allocate rollouts despite heterogeneous gradient informativeness, while softmax policies attenuate high-confidence gradients and excessive updates may destabilize training.

  • Method

    DynaMO derives variance-minimizing rollout allocation using Bernoulli variance and applies gradient-aware advantage modulation using gradient–entropy analysis.

  • Results

    Consistent improvements over strong baselines were demonstrated across multiple reasoning benchmarks and varying LLM scales, with ablations validating both components.

  • Takeaways & Limitations

    DynaMO provides sequence- and token-level controls that target resource allocation, gradient attenuation, and excessive update magnitudes within RLVR optimization.

  • Takeaways & Limitations

    Evaluation covers only the Qwen model family at 1.5B, 7B, and 14B scales, without systematic testing on other architectures.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: https://github.com/GithubX-F/DynaMO-RL.

1 Introduction

RLVR faces inefficient rollout allocation and unstable token-level policy optimization. DynaMO addresses both through variance-minimizing allocation and gradient-aware advantage modulation, with consistent improvements across benchmarks and model scales.

  • Uniform rollout allocation ignores heterogeneous gradient informativeness and the informativeness–noise trade-off across problems.
  • Softmax policies attenuate gradients for high-confidence correct actions, while excessive updates can destabilize training.
  • DynaMO combines variance-minimizing rollout allocation with gradient-aware advantage modulation for sequence- and token-level optimization.
  • The framework proves uniform allocation suboptimal and derives a lightweight proxy for variance-minimizing rollout allocation.
  • Gradient–entropy analysis enables advantage modulation that compensates for attenuation and stabilizes excessive updates.
  • Experiments across six benchmarks and three LLM scales show consistent improvements, with ablations validating each component.

2 Related Works

Related work identifies persistent challenges in computational efficiency and optimization stability. Existing approaches address sample selection, entropy, or clipping, while DynaMO combines sequence-level allocation with token-level control.

  • RLHF and RLVR have achieved substantial success, but computational efficiency and optimization stability remain unresolved challenges.
  • Entropy regularization balances exploration and exploitation, while entropy collapse motivates clipping, reweighting, and entropy-based interventions.
  • Prior sample-efficiency methods select or reorder problems but largely overlook allocating rollout resources according to gradient informativeness.
  • DynaMO’s overview combines dynamic rollout allocation with gradient-aware advantage modulation across sequence and token levels.

3 Preliminaries

The preliminaries formulate autoregressive RLVR policy optimization around response rewards, group-normalized advantages, and a clipped surrogate objective. Policy entropy provides a measure of uncertainty and exploration.

  • RLVR models generate autoregressive responses whose quality is evaluated by a reward function, with training maximizing expected reward.
  • GRPO samples multiple responses per prompt and estimates advantages using group-wise reward normalization.
  • The clipped surrogate objective applies the shared advantage across tokens while using importance ratios and clipping for stability.
  • Policy entropy measures uncertainty in action selection, with higher values indicating greater exploration potential and lower values more deterministic behavior.

4 Methodology

DynaMO combines variance-minimizing dynamic rollout allocation with gradient-aware token-level advantage modulation. The framework allocates resources using Bernoulli variance and balances compensation for attenuated gradients with stabilization against excessive updates.

  • Framework Overview: DynaMO operates at sequence and token levels through dynamic rollout allocation and gradient-aware advantage modulation.The two components provide fine-grained control over resource allocation and policy optimization.
  • Dynamic Rollout Allocation: Uniform rollout allocation is replaced by a variance-minimizing strategy that allocates more rollouts to problems with higher gradient variance.The allocation is formulated under a total rollout-budget constraint and is theoretically derived for policy-gradient estimation.
  • Dynamic Rollout Allocation: Bernoulli variance provides a lightweight proxy for gradient variance, estimated from historical correct and total rollout counts.For binary rewards, the proxy is based on p(1 − p), with balanced correct and incorrect responses producing larger values.
  • Dynamic Rollout Allocation: Water-level allocation enforces minimum and maximum rollout bounds while incrementally updating historical statistics after each training iteration.The implementation allocates budget proportionally to proxy priorities and prevents both insufficient coverage and over-concentration.
  • Gradient Compensation: The softmax policy attenuates gradients for high-confidence correct actions, motivating inverse-entropy compensation for positive-advantage tokens.The compensation function scales inversely with token entropy and moderately amplifies learning signals for confident correct actions.
  • Update Magnitude Stabilization: Entropy changes indicate excessive update magnitudes, so DynaMO applies sigmoid-based stabilization to tokens with large normalized entropy changes.The stabilization factor reduces modulation for unstable tokens, with γ controlling transition sharpness, τ setting the threshold, and λmin defining a lower bound.
  • Integrated Advantage Modulation: The unified advantage modulation preserves learning signals for high-confidence positive actions while damping updates associated with excessive entropy changes.Both mechanisms are combined through a single hyperparameter α in the training objective.

5 Experiments

DynaMO is evaluated across mathematical reasoning benchmarks, model scales, computational budgets, and component settings. Results show consistent gains, with ablations and training analyses supporting the roles of dynamic allocation, gradient compensation, and update stabilization.

  • Main Results: DynaMO consistently outperforms baseline methods across the six-benchmark comparison on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B.The baselines include GRPO and several entropy-intervention methods.
  • Component Analysis: Removing DRA, GC, or UMS degrades average performance, while removing GC and UMS together causes a larger-than-additive degradation.The ablation identifies DRA as especially impactful on Minerva, GC on Olympiad, and UMS as providing cross-benchmark stability.
  • Dynamic Rollout Allocation: Across average budgets of 8 to 32 rollouts per problem, DRA provides stable gains by concentrating resources on problems with peak Bernoulli variance.These problems lie in the capability gap, whereas uniform allocation continues spending resources outside this learning zone.
  • Hyperparameter Sensitivity: The modulation parameter α follows an inverted-U pattern, while broad allocation-bound ranges maintain stable performance.Performance declines both without modulation and at excessive settings.
  • Model Scaling: DynaMO’s advantage over GRPO widens from 1.5B to 7B and 14B models.The evaluation extends to Qwen3-14B across all benchmarks.

6 Case Study

Training-dynamics analysis contrasts DynaMO’s stable optimization with GRPO’s severe fluctuations. The comparison links smoother behavior to adaptive allocation and gradient-aware control.

  • Training Dynamics: DynaMO maintains stable gradient norms and smooth entropy evolution, whereas GRPO exhibits severe gradient spikes and erratic entropy fluctuations.The figure’s shaded regions show the raw data range.

7 Conclusion

DynaMO addresses RLVR’s allocation and policy-optimization challenges through complementary sequence-level and token-level mechanisms. Experiments across benchmarks and model scales report consistent improvements, while ablations validate both mechanisms’ contributions.

  • Conclusion: DynaMO combines variance-minimizing rollout allocation with entropy-informed advantage modulation to improve RLVR optimization.The allocation mechanism targets informative problems, while modulation compensates for attenuated gradients and stabilizes excessive updates.

Limitations

The evaluation is limited to the Qwen model family, although the authors note that DynaMO operates at the algorithmic level.

  • Scope Boundary: Evaluation covers only Qwen models at 1.5B, 7B, and 14B scales, leaving transfer to other architectures and multimodal settings untested.The authors identify broader architecture and multimodal evaluation as future work.

B Gradient-Entropy Relationship: Proof of Expected Gradient Norm Upper Bound

The analysis connects softmax policy geometry to expected gradient magnitude through collision probability and entropy. It shows that high entropy permits larger expected updates, while low entropy attenuates gradients and motivates compensation for confident correct actions.

  • Softmax gradient geometry: The softmax score-function gradient is ∂log πk/∂zi = δik − πi, whose squared norm is analyzed through action-sampling expectations.The derivation begins from the softmax policy and its logit-gradient structure.
  • Collision probability: Collision probability measures how concentrated the policy is, ranging from 1/|V| for uniform sampling to 1 for deterministic sampling.It is the probability that two independent policy samples are identical.
  • Entropy bound: Jensen’s inequality yields an entropy-based upper bound linking expected gradient norm to policy entropy.The proof uses concavity of the logarithm to connect collision probability with Shannon entropy.
  • Entropy bound: High entropy corresponds to vigorous expected updates, whereas near-zero entropy corresponds to vanishing expected gradient magnitude and stable convergence.The bound gives expected gradient norm near 1 at high entropy and near 0 at low entropy.
  • Advantage-weighted dynamics: For advantage-weighted updates, high-confidence tokens with πk ≈ 1 produce small expected update magnitudes, causing gradient attenuation for confident correct actions.This attenuation provides the theoretical basis for gradient compensation in DynaMO.

C Gradient Variance Minimization for Dynamic Rollout Allocation

The allocation analysis formulates rollout assignment as minimizing total policy-gradient variance under a fixed budget. It derives adaptive allocation toward high-variance, high-information problems and describes practical estimation, constraints, and fallback behavior.

  • Variance-minimizing allocation: Uniform rollout allocation is replaced by minimizing total gradient-estimator variance under a fixed rollout budget.The optimization minimizes Σi σ_i^2/n_i subject to the total-budget constraint.
  • Variance decomposition: Gradient variance is decomposed from sequence-level policy-gradient contributions, using approximately orthogonal token-gradient cross terms.The decomposition connects sequence-level variance with token-level gradient terms and their expected squared magnitudes.
  • Variance decomposition: For GRPO, gradient variance positively correlates with reward variance and expected gradient magnitude, identifying informative problems for additional rollouts.This relationship supplies the basis for using observable reward statistics to guide allocation.
  • Bernoulli proxy: Bernoulli variance P_i = p_i(1 − p_i) serves as a computable proxy for σ_i, with accuracy near the variance maximum indicating high-information problems.The proxy estimates binary reward variance and is associated with high policy entropy and multiple explored solution paths.
  • Practical allocation: The water-level algorithm allocates n_i proportionally to P_i while respecting minimum and maximum rollout constraints and fully using the available budget.Residual budget is iteratively redistributed among unconstrained prompts until the budget equality holds.
  • Variance guarantee: Adaptive allocation reduces total variance relative to uniform allocation when gradient variances are heterogeneous across problems.Equality holds only when all σ_i are equal; heterogeneous variances, typical in RLVR, yield substantial variance reduction.
  • Practical allocation: Prompts lacking sufficient historical data receive uniform allocation until reliable variance estimates become available, while later updates track changing problem difficulty.Feedback from new rollouts corrects overestimates and underestimates of historical variance proxies.

F Detailed Description of Benchmarks

The evaluation spans mathematical reasoning benchmarks covering different difficulty levels, formats, and domains. Efficiency analysis reports comparable per-step training time for DynaMO and baselines on Qwen2.5-Math models.

  • Benchmark coverage: The benchmark suite covers competition-level AIME and AMC, curriculum-aligned MATH, and specialized STEM benchmarks including Minerva and OlympiadBench.The tasks span diverse mathematical subfields and problem formats.
  • Benchmark coverage: The benchmarks span secondary-to-graduate difficulty levels, multiple-choice, integer-answer, and open-ended formats, plus pure and applied mathematics.This design supports evaluation across multiple dimensions of mathematical reasoning proficiency.
  • Efficiency analysis: DynaMO maintains comparable per-step training efficiency to baselines on both 1.5B and 7B models.Bernoulli variance estimation, water-level allocation, and token-level modulation add lightweight arithmetic relative to model inference.
  • Efficiency analysis: Figure 6 plots per-step training time on Qwen2.5-Math models using smoothed bold lines and raw light lines.The figure’s visual comparison is between training-time trajectories across model scales.
Loading 2602.19208v2…