On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

1 Introduction

• Prior analyses found sparse RLVR changes but emphasized magnitude; base and final-model histograms for magnitude metrics are nearly identical, limiting their ability to characterize the transformation.The introduction cites high-entropy tokens, KL divergence, and selective gradient updates as prior accounts of sparsity.Source: §1 · p. 1, Figure 1 · p. 2
• Signed token-level Δlog p records probability increases and decreases, reveals a directional signature absent from magnitude metrics, and reaches RLVR-level performance with the fewest token substitutions.The Δlog p histograms are bimodal, while magnitude-based metrics show nearly identical base and RLVR distributions.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Test-time extrapolation selectively amplifies the RLVR policy along the Δlog p direction for reasoning-critical tokens, improving accuracy without additional training.The method extrapolates the learned distributional updates rather than requiring further model training.Source: §1 · p. 2
• Training-time reweighting upweights policy-gradient contributions from low-probability tokens, which correspond to the largest Δlog p, and yields gains over methods such as DAPO across models and benchmarks.The reweighting focuses learning on positions identified as reasoning-critical by Δlog p.Source: §1 · p. 2

• GRPO (Shao et al., 2024), a critic-free PPO variant, estimates advantages across G sampled responses per QA pair and optimizes a clipped policy-ratio objective regularized toward a reference policy.Source: §2 · pp. 2–3
• DAPO (Yu et al., 2025) refines GRPO with clip-higher, dynamic sampling, token-level loss aggregation, overlong punishment, and removal of the KL penalty; it is the primary empirical baseline.Source: §2 · p. 3
• Entropy is calculated for both RLVR and base models, following the observation that RLVR-induced changes are sparse and tend to concentrate on high-entropy tokens (Wang et al., 2025b).Source: §2 · p. 3
• Token-level KL divergence measures distributional shift; Huan et al. (2025) also reported sparse changes, and the analysis includes the reversed KL variant.Source: §2 · p. 3

3.2 Recovering RLVR Performance via Selective Token Replacement

• ∆log p-based replacement matched RLVR accuracy with around 10% of tokens, requiring fewer substitutions than divergence or entropy across model pairs and metric variants.Divergence consistently outperformed entropy, but both required more replacements than ∆log p.Source: Figure 2 · p. 4, §3.2 · p. 5

3.1 Statistical Analysis: Directional vs. Magnitude-Based Metrics

• Across ORZ, DAPO, and UniReason outputs on AIME-24, metric distributions were near zero; entropy and KL overlapped across models, while ∆log p showed distinct signed tails.The pattern held across tested model pairs and entropy/divergence variants, with ∆log p consistently bimodal.Source: §3.1 · p. 4

3.2 Recovering RLVR Performance via Selective Token Replacement

• Replacing 5-30% of base-model tokens with metric-selected RLVR tokens recovered final RLVR accuracy, while random replacement improved performance more slowly.This indicates that RLVR-modified tokens are sparse but disproportionately important for reasoning.Source: §3.2 · p. 5

3.3 A Gradient-Based Explanation for the Sparse Updates

• Low-probability tokens receive larger gradients and account for most gradient mass; high-∆log p tokens are also low-probability in both models, with probability increasing under RLVR.This links sparse RLVR changes to concentrated updates on rare, high-impact tokens.Source: §3.3 · p. 6
• Filtering low-probability tokens during Qwen2.5-Math-7B DAPO rollouts impaired performance: even top-p=0.95 fell below top-p=1.0, and stricter filtering degraded it further.The intervention supports the importance of low-probability tokens to RLVR reasoning gains.Source: §3.3 · p. 6

4.1 Test-Time Enhancement via Extrapolation

• The method adds γ times the learned Δlog p to RLVR log probabilities, then selectively samples from the extrapolated policy at positions gated by large negative Δlog p.Because most positions have negligible Δlog p, selective gating aims to avoid distorting well-calibrated tokens.Source: §4.1 · p. 7
• On AIME-24, selective extrapolation achieves higher Avg@32 than the RLVR baseline across ORZ-32B, DAPO-32B, and UniReason-14B, while selective replacement does not surpass it.Replacement and extrapolation use matched selection criteria; the difference is the extrapolation.Source: Figure 4 · p. 8, §4.1 · p. 8
• In a simplified tabular softmax setting with natural policy gradient updates, the analysis indicates extrapolation along the learned difference direction can improve expected reward.The proof relies on idealized monotonic updates along the reward direction; the authors therefore motivate selective extrapolation at positions with significant Δlog p.Source: §4.1 · p. 8
• Extrapolating directly on DAPO-32B improves AIME-24 performance as the intervention ratio rises to a moderate level, after which gains plateau.The pattern is consistent with amplifying a limited set of reasoning-critical tokens rather than intervening aggressively.Source: Table 1 · p. 8, §4.1 · p. 8

4.2 Training-Time Enhancement via Advantage Reweighting

• Training-time reweighting modifies DAPO’s advantage to prioritize low-probability tokens, which are associated with high Δlog p and identified as important for RLVR training.The reweighting strength is controlled by α.Source: §4.2 · pp. 8–9
• Across Qwen2.5-Math-7B and Qwen3-8B-Base, low-probability-token reweighting consistently improves reasoning accuracy (Avg@32) across tested datasets, while average Pass@16 also increases over the DAPO baseline.Source: §4.2 · p. 9
• On Qwen2.5-Math-7B, low-probability-token reweighting achieves the best overall Avg@32 and Pass@16 among the compared methods, with response length steadily increasing during training.Dominate has lower training entropy under its tighter clip-higher ratio, which the authors link to reduced Pass@k.Source: Table 3 · p. 9, §4.2 · pp. 9–10, Figure 5 · p. 10

5 Related Work

• LLM reinforcement learning shifted from RLHF alignment (Ouyang et al., 2022) toward reasoning models, with this study adopting DAPO as its RLVR baseline (Yu et al., 2025).OpenAI o1 (Jaech et al., 2024) and DeepSeek R1 helped inspire later industrial and academic reasoning systems.Source: §5 · p. 10
• Prior RLVR analyses found sparse updates but emphasized change magnitude; this paper extends update direction to the full distributional shift and proposes extrapolating along it.Earlier work quantified entropy changes (Cui et al., 2025), token gradient norms (Yang et al., 2025b), and token-replacement effects (Meng et al., 2026); sparsity was measured using entropy, KL divergence, and sparse gradient norms.Source: §5 · p. 10

6 Conclusion

• Δlog p identifies sparse, reasoning-critical RLVR updates better than magnitude-based metrics, and extrapolation and reweighting improve reasoning performance across settings.The authors frame update direction as a principle for diagnosing and improving RLVR.Source: §6 · p. 10
• The extrapolation method requires two models and adds hyperparameters; proposed future work includes parameter-efficient finetuning and more adaptive threshold-strength selection.The authors also suggest evaluating reweighting at different model scales or combining it with adaptive training techniques.Source: §6 · p. 11
• The reweighting approach remains to be evaluated across different model scales and in combination with other adaptive training techniques.These are identified as directions for future work.Source: §6 · p. 11

A.1 Implementation Details

• The experiments use Qwen-series RLVR models: ORZ and DAPO finetuned from Qwen2.5-32B, and UniReason finetuned from Qwen3-14B-Base.ORZ uses PPO, DAPO uses DAPO, and UniReason uses GRPO (Hu et al., 2025a; Yu et al., 2025; Huan et al., 2025).Source: §A.1 · p. 15
• Replacement performance is evaluated on AIME-24 with each model’s default chat template and a prompt requesting step-by-step reasoning and a boxed final answer.Sampling uses top-p=0.7, temperature=1.0, max-length=20k, and 32 responses per question; correctness is verified with Math-Verify and averaged as Avg@32.Source: §A.1 · p. 15
• Selective replacement is controlled by threshold τ, while extrapolation strength is controlled by γ; the default ϵ_high=0.28 caused unstable training.The passage says the chosen value follows the recommendation in Yang et al. (2025b).Source: §6 · p. 10, §A.1 · p. 15

A Selective Token Replacement & Extraploation

• For Fig. 4, Selective Extrapolate and Selective Replace share the same hyperparameters for each model, summarized in Table 4.The supplied table passage gives its scope but not hyperparameter values.Source: §A.1 · p. 15, Table 4 · p. 15

Additional Experiments

• On AIME-24, Δlog p reaches RLVR-level performance with fewer replacements than divergence or entropy, ranking highest for identifying reasoning-critical tokens.The ablation also evaluates RLVR-model entropy and two KL-divergence variants alongside base-model entropy, KL divergence, and log-probability difference.Source: §A.2 · p. 15, Figure 6 · p. 16
• At an approximately 8% replacement rate, entropy favors logical transition words, whereas divergence and Δlog p select more specific mathematical tokens, including more math symbols.The authors suggest these mathematical tokens may improve reasoning more efficiently, consistent with entropy’s inferior performance.Source: §A.2 · pp. 15–16, Figure 7 · p. 16
• Across problems, accuracy generally rises with increasing replacement before fluctuating, while inherently difficult problems remain at zero and replacement-only performance is capped by the RLVR model.A few problems initially lose accuracy at low replacement ratios and improve as replacement increases.Source: §A.2 · p. 16, Figure 8 · p. 17
• For some initially harmed problems, unfamiliar RL-replaced tokens disrupt coherent generation; increasing replacement then guides generation more strongly with RL tokens, allowing performance to recover and improve.The passage gives DAPO problem 0 as an example of an initial accuracy drop followed by improvement.Source: §A.2 · pp. 16–17

A.3 Hyperparameter Sensitivity Analysis

• The grid search tests γ values of 0.05 and 0.1 and varies τ by model, using a denser τ grid for UniReason because fewer replacements are needed to reach RLVR performance.DAPO and ORZ use τ ∈ {−0.5, −0.4, −0.3, −0.2, −0.1}; UniReason uses τ ∈ {−0.5, −0.45, −0.4, −0.35, −0.3}.Source: §A.3 · pp. 17–18
• Once replacement matches RLVR performance, further increases in replacement provide little to no added benefit because performance is bounded by the RLVR model.In contrast, proper test-time extrapolation can exceed RLVR by 1–3 points.Source: §A.3 · p. 18
• 1–3 points: Across nearly all models and hyperparameter settings, extrapolation outperforms replace-only replacement and can exceed RLVR performance without additional training.The comparison is reported for the selective extrapolation experiment in Table 5.Source: Table 5 · p. 18, §A.3 · p. 18

Hyperparameters Setting

• The DAPO-based setup uses clip ratios ϵlow = 0.2 and ϵhigh = 0.28, learning rate 1e-6, 10-step warmup, and 16 gradient updates per RLVR step.Each step uses 512 prompts with 16 responses each, in mini-batches of 32 prompts; maximum generation lengths and overlong thresholds are 8k and 4k for Qwen2.5-Math-7B, and 20k and 16k for Qwen3-8b-base.Source: B RLVR Training Details · p. 18
• Reweighting α is 0.2 for Qwen2.5 and 0.1 for Qwen3; α is 0.1 for ˜Adom_i,t and 0.01 for ˜APPL_i,t, with ϵhigh adjusted to 0.24.The latter settings follow recommended values by Deng et al. (2025) and Yang et al. (2025b).Source: B RLVR Training Details · p. 18

Reproducibility Analysis

• Across four Qwen2.5-Math-7B reweighting runs, learning curves converged consistently and every trial reached or surpassed the performance reported in Tab. 3.Run 1 is the run reported in Fig. 5.Source: Figure 9 · p. 19, B RLVR Training Details · p. 19

C Performance beyond Pure-Math Reasoning Tasks

• To assess reasoning generalization on non-math tasks beyond math-focused datasets, the authors use Minerva, which contains 272 undergraduate-level STEM problems spanning subjects such as Chemistry and Astronomy6.Source: C Performance beyond Pure-Math… · p. 19
• Reweighting-trained models outperform baselines on Minerva reasoning accuracy while maintaining comparable or improved Pass@k scores.The evaluation uses the same sampling parameters as other evaluations; Table 6 reports RLVR-trained model performance on Minerva.Source: C Performance beyond Pure-Math… · p. 19, Table 6 · p. 19
• Test-time extrapolation consistently improves Avg@8 accuracy over the RLVR model on Minerva for evaluated 14B–32B models.Avg@8 is reported because Minerva has around seven times as many questions as AIME24; hyperparameter-grid results also consistently outperform replacing with πRL only.Source: C Performance beyond Pure-Math… · p. 19, Figure 10 · p. 20, Table 7 · p. 20

D Proofs

• The appendix derives the gradient of DAPO for an unclipped token under a softmax-parameterized policy and expresses its gradient’s ℓ1-norm.The derivation omits context notation and treats the token as vocabulary index k.Source: D Proofs · pp. 20–21
• Theorem 4.1’s proof shows that the extrapolation direction has a nonnegative expected-reward directional derivative, with equality only when rewards are constant.The proof uses the zero expected advantage and the Chebyshev sum inequality.Source: D Proofs · pp. 21–22
• Under the bandit assumption, advantage and extrapolation-direction values are ordered like rewards; induction establishes this ordering for the direction at every iteration.The proof begins from the NPG update and extends the ordering from iteration t to t + 1.Source: D Proofs · p. 21

E Statistical Comparison of Different Metrics

• The evaluation uses ORZ, DAPO, and UniReason with their base counterparts, generating 32 responses per AIME-24 question at top-p=0.7 and temperature=1.0.Source: E Statistical Comparison of Different… · p. 22
• The analysis compares entropy, KL divergences, and log-probability difference on both base and RLVR generations, and measures Δlog p-bin probabilities on DAPO generations.Source: E Statistical Comparison of Different… · p. 22
• Top-100 high-Δlog p tokens include explicit reasoning actions such as “combine,” “break,” and “simplify,” and logical transitions such as “wait,” “think,” and “step.”The passage says this vocabulary suggests that RLVR models learned to construct more effective reasoning processes.Source: E Statistical Comparison of Different… · p. 22, Figure 11 · p. 23

F The Use of Large Language Models

• The authors used LLMs only to polish some pre-existing paper language; they reviewed suggested modifications for accuracy and alignment with the original meaning.Source: F The Use of Large Language Models · p. 22

E Statistical Comparison of Different Metrics

• The Δlog p distributions are bimodal, with a positive tail for RLVR generations and a negative tail for base generations, unlike the nearly identical magnitude-based metric distributions.The comparison covers entropy and KL divergences; Figures 12–15 show the log-probability-difference, divergence, and entropy histograms.Source: E Statistical Comparison of Different… · p. 22, Figure 12 · p. 24, Figure 13 · p. 25