PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation

Kexin Huang, Haoming Meng, Junkang Wu, Jinda Lu, Chiyu Ma, Ziqian Chen, Xue Wang, Bolin Ding, Jiancan Wu, Xiang Wang, Xiangnan He, Guoyin Wang, Jingren Zhou

arXiv:2603.22117v1cs.LGcs.AI

TL;DR

Existing RLVR analyses identify sparse updates but largely overlook their direction, motivating a signed token-level Δlog p diagnostic. The paper validates this diagnostic with statistical analysis and token replacement, then uses it for test-time extrapolation and training-time reweighting, which improve reasoning performance across settings.

  • Problem

    Existing analyses show RLVR updates are sparse but primarily measure their magnitude, leaving their direction insufficiently characterized.

  • Method

    The paper analyzes signed token-level Δlog p, validates it through statistical comparisons and token-replacement interventions, and develops test-time extrapolation plus training-time advantage reweighting.

  • Results

    Δlog p more effectively identifies sparse, reasoning-critical updates than magnitude-based metrics, while both proposed strategies improve reasoning performance across different settings.

  • Takeaways & Limitations

    Update direction is a useful principle for diagnosing RLVR effects and guiding reasoning enhancement.

  • Takeaways & Limitations

    The extrapolation method requires two models and adds hyperparameters, while the reweighting approach remains to be evaluated across different model scales or with other adaptive training techniques.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the \textbf{magnitude} of these updates, largely overlooking their \textbf{direction}. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $Δ\log p$ between the base and final RLVR models. Through statistical analysis and token-replacement interventions, we demonstrate that $Δ\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (\eg divergence or entropy). Building on this insight, we propose two practical applications: (1) a \textit{test-time extrapolation} method that amplifies the policy along the learned $Δ\log p$ direction to improve reasoning accuracy without further training; (2) a \textit{training-time reweighting} method that focuses learning on low-probability (corresponding to higher $Δ\log p$) tokens, which improves reasoning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

1 Introduction

RLVR updates are sparse, but prior analyses largely measure only their magnitude rather than direction. This paper introduces signed token-level Δlog p to identify reasoning-critical changes and exploit them at test time and during training.

  • Motivation: Prior studies find RLVR-induced changes sparse, but primarily characterize their magnitude through entropy, KL divergence, or selective gradient updates.These analyses largely overlook whether probability mass shifts toward or away from a token.
  • Directional diagnostic: Signed token-level Δlog p measures how RLVR shifts probability mass, with positive values indicating increased token probabilities and negative values indicating decreased probabilities.The metric is used to distinguish tokens favored by RLVR from those favored by the base model.
  • Directional diagnostic: Only Δlog p shows a directional shift distinguishing RLVR from base-model generations, whereas magnitude-based metrics concentrate near zero without that separation.Figure 1 compares token-level metric distributions on responses generated by the two models.
  • Validation: Token replacement experiments show that selecting positions with Δlog p recovers RLVR performance with fewer replacements than alternative metrics.The intervention replaces base-model tokens with RLVR choices at metric-selected positions.
  • Applications: The paper proposes selective test-time extrapolation along the learned Δlog p direction and training-time reweighting that emphasizes low-probability tokens.The two strategies target reasoning-critical positions identified through directional analysis.
  • Applications: Training-time reweighting improves reasoning performance across diverse benchmarks and models, including relative gains over DAPO.The method upweights contributions from low-probability tokens associated with larger Δlog p.

2 Preliminaries

The paper situates its analysis within critic-free policy optimization methods for RLVR, especially GRPO and its refinement DAPO. It also compares token-level entropy and divergences as alternative measures of RLVR-induced changes.

  • Group Relative Policy Optimization: GRPO adapts PPO-style policy gradients to LLM training by eliminating the need for a separate critic model.For each question-answer pair, GRPO samples a group of responses and estimates advantages comparatively within that group.
  • Group Relative Policy Optimization: GRPO optimizes its policy using a clipped objective based on an importance-sampling ratio and response-level advantages.The clipping range constrains the importance-sampling ratio during optimization.
  • Policy objectives: The KL divergence term regularizes the policy toward a reference policy when included in the objective.The preliminaries describe D_KL(π_θ||π_ref) as the policy-to-reference constraint.
  • Dynamic Sampling Policy Optimization: DAPO is presented as a state-of-the-art critic-free RLVR algorithm that further refines GRPO with several sampling and objective-design techniques.Its listed techniques include clip-higher, dynamic sampling, token-level loss aggregation, overlong punishment, and removal of the KL penalty.
  • Dynamic Sampling Policy Optimization: The paper adopts DAPO as the primary baseline for its empirical analysis.This choice follows DAPO’s stated success among the discussed RLVR algorithms.
  • Token-level metrics: Token-level entropy and KL divergence are included as magnitude-oriented metrics for comparing RLVR and base-model distributions.Entropy is computed for both models, while prior work uses KL divergence to quantify distributional shift and sparsity.

3 Dissecting the Token-Level Changes Introduced by RLVR

RLVR changes are sparse but concentrated in reasoning-critical tokens, and signed Δlog p identifies these directional shifts more precisely than magnitude-only metrics. Token replacement and gradient analyses connect this precision to RLVR’s emphasis on rare, low-probability tokens.

  • Directional token changes: Statistical analysis compares entropy, KL divergence, and Δlog p across several RLVR-base model pairs using generated AIME-24 tokens.The analysis covers ORZ, DAPO, and UniReason model pairs.
  • Directional token changes: RLVR-induced changes are sparse, and Δlog p captures their directional probability shifts whereas entropy and divergence do not.Positive and negative Δlog p tails distinguish tokens favored by RLVR from those favored by the base model.
  • Selective token replacement: Selective replacement samples from πBase and replaces metric-selected tokens with πRL tokens at matched replacement rates.Metric-specific thresholds control replacement rates for fair comparisons, with random replacement as a baseline.
  • Selective token replacement: 5-30% selective replacement can recover RLVR-level accuracy, whereas random replacement improves much more slowly.This shows that a minority of RLVR-modified tokens are disproportionately important for reasoning.
  • Selective token replacement: Around 10% of tokens suffice for Δlog p-based replacement to reach RLVR accuracy, outperforming divergence and entropy in precision.The stable ranking is Δlog p > divergence > entropy across model pairs and metric variants.
  • Gradient-based explanation: RLVR gradients concentrate on rare, low-probability tokens, which correspond to the high-Δlog p tokens receiving the most consequential updates.Low-probability tokens receive disproportionately large gradient updates, and high-Δlog p bins contain lower-probability tokens in both models.
  • Conclusion: RLVR gains arise from targeted, high-impact changes to a minority of tokens, with Δlog p pinpointing these sparse changes.The conclusion contrasts targeted modifications with a global distribution shift.

4 Exploiting RLVR’s Directional Updates to Boost Reasoning

The paper exploits RLVR’s directional updates in two ways: selective test-time extrapolation and training-time reweighting of low-probability tokens. Both strategies use Δlog p to target reasoning-critical updates and improve reasoning performance.

  • Test-Time Enhancement via Extrapolation: Selective extrapolation amplifies the RLVR distribution along the learned Δlog p direction on selected critical tokens.The method modifies the RLVR distribution using γ to control extrapolation strength and z(·) as a log-partition function.
  • Test-Time Enhancement via Extrapolation: The extrapolation acts like token-level reward-guided decoding, with Δlog p reweighting probabilities toward better reasoning.The paper frames Δlog p as a token-level reward that encourages better reasoning.
  • Test-Time Enhancement via Extrapolation: Selective gating targets positions with large negative Δlog p because most positions have negligible changes and global intervention could distort calibrated tokens.The criterion fτ determines where extrapolation is applied, while selected positions use the extrapolated distribution.
  • Test-Time Enhancement via Extrapolation: On AIME-24, Selective Extrapolate achieves higher Avg@32 than πRL across ORZ-32B, DAPO-32B, and UniReason-14B under matched gates.Selective Replace matches but does not surpass the RL baseline, whereas extrapolation provides incremental reasoning gains.
  • Test-Time Enhancement via Extrapolation: Amplifying πRL improves AIME-24 performance up to a moderate intervention ratio, after which gains plateau.This supports selectively amplifying a limited set of reasoning-critical tokens rather than intervening aggressively.
  • Training-Time Enhancement via Advantage Reweighting: Training-time reweighting increases the policy-gradient advantage of low-probability tokens, which are associated with high Δlog p and are treated as reasoning-critical.The method changes only the advantage term while keeping the remaining DAPO recipe fixed, and evaluates AIME-24, AIME-25, and AMC.
  • Training-Time Enhancement via Advantage Reweighting: The reweighting method consistently improves reasoning accuracy across Qwen2.5-Math-7B and Qwen3-8B-Base on the tested benchmarks without reducing exploration ability.Average Pass@16 also increases over the DAPO baseline.
  • Training-Time Enhancement via Advantage Reweighting: Against alternative reweighting methods, direct amplification of low-probability tokens achieves the best overall Avg@32 and Pass@16 on Qwen2.5-Math-7B.Its training curves also show higher reasoning accuracy and a steady increase in response length.

5 Related Work

Research on RLVR has progressed from analyzing high-level reasoning behaviors to quantifying token-level changes, including entropy and gradient-related measures. This paper adopts DAPO as its baseline RLVR algorithm.

  • The paper uses DAPO as its baseline RLVR algorithm.
  • Recent RLVR studies analyze its effects at both the behavioral and token levels.Token-level analyses include entropy change and gradient-norm measurements.

6 Conclusion

The paper identifies signed log-probability change as a directional diagnostic that more effectively locates reasoning-critical RLVR updates than magnitude-based metrics. It then applies this diagnostic to test-time extrapolation and training-time reweighting, while acknowledging computational and hyperparameter limitations.

  • Δlog p more effectively identifies sparse, reasoning-critical updates than divergence or entropy.
  • Test-time extrapolation amplifies directional updates, while training-time reweighting emphasizes low-probability tokens highlighted by Δlog p.
  • The extrapolation method requires two models and introduces additional hyperparameters.
  • Future work could reduce extrapolation cost through parameter-efficient fine-tuning and evaluate reweighting across model scales.

Contributions

The paper’s empirical work uses publicly available Qwen-series RLVR models and corresponding bases, evaluates selective replacement on AIME-24, and controls extrapolation through threshold and strength hyperparameters.

  • Models: Experiments use ORZ, DAPO, and UniReason RLVR models with corresponding Qwen-series base models.The models were trained with PPO, DAPO, and GRPO respectively.
  • Sampling settings: Selective replacement is evaluated on AIME-24 using each model’s default chat prompt template.
  • Sampling settings: Responses are sampled with top-p=0.7, temperature=1.0, max-length=20k, and 32 samples per question.Correctness is averaged over 32 samples as Avg@32 after extraction and Math-Verify checking.
  • Hyperparameters: Extrapolation uses a threshold τ to select replacements and γ to control extrapolation strength.
  • Hyperparameters: Table 5 reports hyperparameter sensitivity for selective extrapolation, distinguishing reported extrapolation values from token-replacement endpoints.

A.2 Additional Experiments

Additional experiments compare token-selection metrics, analyze which tokens are replaced, track per-problem accuracy, and test extrapolation sensitivity. Across these analyses, Δlog p requires fewer replacements and extrapolation exceeds replace-only performance under suitable settings.

  • Metric comparison: Additional metrics include base-model entropy, KL divergence, RLVR-model entropy, and KL-divergence variants alongside Δlog p.
  • Metric comparison: Additional criteria approach RLVR performance but require more replacement than Δlog p.
  • Selected tokens: At an approximately 8% replacement rate, entropy favors logical transition words, while Δlog p selects tokens with a higher proportion of math symbols.The text connects entropy’s inferior performance with the possibility that selected mathematical tokens are more efficient for reasoning improvement.
  • Per-problem accuracy: Some problems remain at zero accuracy across replacement ratios, while accuracy generally increases for other problems as replacement rises.
  • Per-problem accuracy: Small numbers of unfamiliar RL-replaced tokens can initially disrupt coherence before stronger RL guidance restores and improves performance.
  • Extrapolation sensitivity: Across nearly all models and hyperparameter settings, extrapolation outperforms replace-only and can exceed RLVR performance by 1–3 points without additional training.Once replacement alone matches RLVR, further replacement provides little additional benefit because performance is bounded by the RLVR model.

B RLVR Training Details

The RLVR experiments use specified training configurations and assess whether the reweighting method converges reproducibly across independent runs.

  • Training configuration: RLVR training uses the open-sourced DAPO recipe with double clip ratios, a 1e-6 learning rate, and 10-step warmup.Each step processes 512 prompts with 16 responses each and produces 16 gradient updates.
  • Reweighting configuration: The reweighting parameter α is set to 0.2 for Qwen2.5 and 0.1 for Qwen3.
  • Reproducibility analysis: All four runs reached or surpassed the performance levels reported in Table 3, indicating consistent convergence and performance.

C Performance beyond Pure-Math Reasoning Tasks

The paper evaluates reasoning beyond math-focused training using Minerva, finding that reweighting and test-time extrapolation retain advantages on this broader STEM benchmark.

  • Benchmark and motivation: Minerva contains 272 undergraduate-level STEM problems across subjects including Chemistry and Astronomy.The evaluation tests generalization beyond math-focused datasets.
  • Reweighting results: Reweighting-trained models outperform baselines on Minerva reasoning accuracy without reducing exploration, as shown by comparable or improved Pass@k scores.
  • Test-time extrapolation: Test-time extrapolation consistently improves accuracy over the RLVR model on Minerva for evaluated 14B–32B models.The paper reports Avg@8 because Minerva has around seven times more questions than AIME24.
  • Test-time extrapolation: The Minerva extrapolation settings and Avg@8 results are reported in Table 7, with tuned values marked by an asterisk.

D Proofs

The proofs analyze how advantage and directional terms are ordered under simplified policy-update assumptions, establishing conditions for directional extrapolation and reward alignment.

  • Directional derivative: The directional derivative analysis yields a nonnegative gradient expression involving policy probabilities, advantages, and directional terms.
  • Directional derivative: For a softmax policy, the proof derives the relevant gradient relations in logit coordinates to analyze directional derivatives.
  • Ordering argument: Under the bandit setting, advantage ordering matches reward ordering: At(x,y1) > At(x,y2) exactly when Rx,y1 > Rx,y2.
  • Ordering argument: Induction shows that the directional term dt_x,y preserves the same ordering as reward Rx,y across iterations.
  • Equality condition: Equality holds only when rewards are constant across actions, using the zero expected-advantage property and the Chebyshev sum inequality.

E Statistical Comparison of Different Metrics

Statistical comparisons show that signed Δlog p distinguishes RLVR-generated from base-generated text, whereas divergence and entropy distributions remain largely insensitive to the generating model.

  • Empirical setup: The analysis compares entropy, KL divergences, and Δlog p across ORZ, DAPO, UniReason, and their base models using AIME-24 generations.Each model generates 32 responses per question with top-p=0.7 and temperature=1.0.
  • Metric distributions: Δlog p distributions are bimodal, with a positive tail on RLVR generations and a negative tail on base-model generations.
  • Metric distributions: Divergence and entropy distributions are nearly identical regardless of whether the output was generated by the RLVR or base model.
  • Token interpretation: The top-100 high-Δlog p tokens primarily represent explicit reasoning actions and logical transitions.Examples include “combine,” “break,” “simplify,” “wait,” “think,” and “step.”
Loading 2603.22117v1…