PaperlayerView exact revision on arXiv

Source-linked AI summary

GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning

Outongyi Lv, Yuanwei Zhang, Xiaoqun Zhang

arXiv:2608.30632v1cs.CLcs.AIcs.LG

TL;DR

The paper addresses the limitation of entropy alone as a measure of token importance in RLVR, because reward variation across answers can alter token contributions. It introduces GMTS, which uses a gradient-magnitude proxy for token selection, and reports consistent improvements over ETS across reasoning tasks and model scales.

  • Problem

    Entropy correlates with gradient magnitude within an answer but does not consistently capture token importance across answers with varying reward signals.

  • Method

    GMTS leverages the entropy-gradient relationship to approximate gradient-magnitude rankings for selecting tokens during RLVR training.

  • Results

    GMTS consistently improves performance over ETS across multiple selection ratios, model scales, and evaluation domains.

  • Takeaways & Limitations

    GMTS provides an effective and efficient token-selection choice for RLVR and can be integrated into existing frameworks with minimal computational overhead.

  • Takeaways & Limitations

    The theoretical explanation for why GMTS outperforms ETS remains incomplete.

Abstract

from arXiv · show

Reinforcement learning (RL), particularly RL with Verifiable Rewards (RLVR), has recently emerged as a central paradigm for enhancing large language models' (LLMs) reasoning abilities, demonstrating remarkable effectiveness across reasoning tasks. Recent studies suggest that high-entropy tokens play an exceptionally important role in model training, since training with only the highest 20% entropy tokens yields significant performance gains. However, why such high-entropy tokens are beneficial remains insufficiently understood. In this work, we find that although high-entropy tokens within one answer tend to correlate with large gradient magnitude, entropy alone fails to consistently reflect token importance across different answers, considering the variations in the answer-level reward signals. Based on this observation, we introduce the Gradient Magnitude-based Token Selection (GMTS) method to quantify token importance, which leverages the entropy-gradient connection to approximate gradient-magnitude rankings for token selection. We find that training on the top 20% tokens ranked by GMTS consistently outperforms entropy-based token selection across three reasoning domains and various model sizes, suggesting that GMTS provides a more fine-grained estimate of token contribution for RLVR training.

1 Introduction

RLVR improves LLM reasoning, but entropy-based token selection does not consistently measure token importance across answers with differing reward signals. GMTS addresses this gap by using gradient-magnitude information and consistently outperforms entropy-based baselines across reasoning domains and model scales.

  • RLVR has become an effective paradigm for advancing LLM reasoning abilities through scalable reinforcement learning with verifiable rewards.
  • High-entropy tokens are widely used for RLVR selection, including selecting the top 20% entropy tokens for training.
  • Within an answer, high entropy correlates with large gradient magnitude, but entropy alone fails across answers when reward signals and sample characteristics vary.
  • GMTS consistently outperforms baselines across three reasoning domains and multiple model scales, with approximately 1-3 percentage point gains on math reasoning benchmarks.
  • GMTS quantifies token importance using the entropy-gradient relationship while addressing entropy’s limitations across different answers.

2 Preliminaries

This section defines autoregressive LLM generation and predictive-distribution entropy, then reviews GRPO and DAPO as representative RLVR methods. GRPO estimates response advantages without a value model, while DAPO adds practical modifications for sampling and training stability.

  • 2.1 LLM Formulation and Notations: An LLM autoregressively generates an output sequence by sampling or selecting each token from a distribution conditioned on the query and prior tokens.
  • 2.1 LLM Formulation and Notations: Entropy is defined for the model’s predictive distribution at each generation step.
  • 2.2 RLVR Methods: GRPO estimates advantages from groups of sampled responses, eliminating the need for a separate value model.
  • 2.2 RLVR Methods: GRPO assigns each response-level advantage to all tokens in that response and optimizes a clipped objective with KL regularization.
  • 2.2 RLVR Methods: DAPO removes the KL penalty, discards uninformative groups through dynamic sampling, and uses token-level loss averaging with a larger upper clipping threshold.

3 Methodology

GMTS selects RLVR tokens using an estimate of their gradient magnitude rather than entropy alone. The method is motivated by entropy’s within-answer relationship to gradient size, while accounting for answer-level advantages, clipping, and KL-related effects that alter token contributions.

  • GRPO and DAPO assign the same response-level advantage to every token, ignoring unequal token contributions to policy updates.
  • Empirical analysis: The empirical analysis compares entropy and gradient measures across MATH-12K, MATH-500, and KodCode settings, including 1.5B and 7B models.
  • Gradient of each token: The effective coefficient ω_i,t(θ) scales each token’s policy gradient using advantage, clipping, and KL regularization signals.
  • Token gradient magnitude and entropy: Within an answer, token-level log-probability gradient magnitude has an approximately linear relationship with token entropy.
  • Gradient magnitude token selection: Entropy alone can misrepresent token contribution because gradient coefficients vary across answers with different advantages and clipping-induced gradient magnitudes.
  • Gradient magnitude token selection: GMTS ranks tokens with the score |E_i,t · ω_i,t(θ)| and retains the top-ρ fraction above threshold τ_ρ for the training gradient.

4 Numerical Experiments

Experiments evaluate GMTS across reasoning domains, model scales, backbones, and selection settings. GMTS generally outperforms ETS and baselines, while its benefits remain robust across ratios and tasks.

  • Experimental Setup: GMTS is implemented with ETS across GRPO and DAPO in the verl framework, primarily using the top 20% of tokens.Experiments cover MATH, CODE, and COMMONSENSE, with primary evaluations at 1.5B and 7B and additional 8B experiments.
  • MATH Results: GMTS Top (20%) outperforms ETS across DAPO and GRPO on Qwen2.5-math at both 1.5B and 7B scales.Reported gains are +1.55, +1.30, +1.33, and +3.41 for 1.5B-DAPO, 1.5B-GRPO, 7B-DAPO, and 7B-GRPO, respectively.
  • MATH Results: GMTS delivers an overall average gain of 1.85% over ETS on Qwen3-8B across six math-reasoning benchmarks.Improvements include +5.21 on AIME2024 and +3.75 on AIME2025.
  • CODE and CS Results: GMTS achieves consistent gains in CODE and COMMONSENSE evaluations across model sizes and DAPO or GRPO backbones.Code gains range from +0.69 to +1.90, while commonsense gains range from +0.67 to +1.53 across the reported configurations.
  • Ablation Study: GMTS shows linear and monotonic behavior across Qwen2.5-math and Qwen2.5-coder at 1.5B and 7B scales.The reported monotonicity indicates that gradient-magnitude-based selection preserves token-contribution ordering.
  • Ablation Study: GMTS outperforms ETS in nine of ten tested selection-ratio configurations, but performs worse than ETS under bottom selection.Bottom-selection average drops are -1.08 and -1.28 at the 80% and 90% ratios, respectively.

5 Conclusion

The paper concludes that GMTS uses a gradient-magnitude proxy to address entropy-based token selection’s limitations in RLVR. Across domains, scales, and ratios, it consistently improves over ETS.

  • Conclusion: GMTS addresses entropy-based token selection’s limited cross-answer reliability by using a gradient-magnitude proxy for token importance.Entropy correlates with gradient magnitude within an answer, but token contributions can vary across answers because reward signals differ.
  • Conclusion: GMTS consistently improves over ETS across multiple selection ratios, model scales, and evaluation domains.The conclusion presents GMTS as an effective and efficient choice for RLVR based on the reported empirical evidence.

Limitations

The paper identifies unresolved theoretical and scalability limitations for GMTS. Its theoretical basis is only partially understood, and evaluation has not yet covered larger 14B and 32B models.

  • Theoretical limitations: GMTS's theoretical foundation remains only partially understood despite extensive empirical evaluations.The paper provides preliminary evidence linking token entropy and gradient magnitude but lacks a rigorous theoretical justification for GMTS's advantage over ETS.
  • Theoretical limitations: A rigorous explanation for why GMTS outperforms ETS and a deeper analytical characterization remain future-work directions.
  • Scalability boundary: Due to computational constraints, GMTS has not been evaluated on larger 14B and 32B models.The authors leave comprehensive evaluation at these scales for future work, although existing results suggest potential extension.

A.1 More Experimental Results

Additional experiments examine whether GMTS selects more beneficial tokens than ETS and whether its advantage persists across selection ratios. GMTS outperforms ETS in 9 of 10 GRPO/DAPO settings.

  • Reported results: Complete numerical results for Section 4.3 are reported in Tables 6 and 7.
  • Selection-ratio results: GMTS outperforms ETS in 9 out of 10 GRPO/DAPO settings across different top-selection ratios.This result indicates that GMTS generally remains superior to ETS in the reported selection-ratio experiments.
  • Selection comparison: The Bottom selection experiment tests whether GMTS's top-ranked tokens contain more training-beneficial information than ETS-selected tokens.

A.2 Derivation of token gradient

The derivation defines token gradient magnitude and entropy for the model's next-token distribution, then relates these quantities through probability-based expressions and a residual entropy term.

  • Proposition 3.1: Proposition 3.1 defines G_t as the ℓ2-norm of the log-probability gradient with respect to logits and E_t as policy entropy.
  • Probability notation: The derivation sets the generated-token probability to p_t,i and introduces ε as one minus that probability.
  • Evaluation context: Tables 6 and 7 evaluate ETS and GMTS with DAPO or GRPO on Qwen2.5-math-1.5B using average@16 accuracy, including different selected ratios in Table 7.
  • Residual distribution: The residual distribution q_t,k is expressed using p_t,k and ε as part of the gradient derivation.
  • Entropy derivation: The entropy derivation considers the entropy of the renormalized residual distribution after removing the generated token's probability.

A.3 Training Details

The appendix reports implementation settings for reproducible RLVR experiments, including fixed hardware and seed choices, model-specific sequence limits, batching, rollout, and dialogue-template configurations.

  • General setup: All experiments use training seed 0 on eight NVIDIA H100 GPUs with 80 GB of memory each.
  • Math models: Qwen2.5-math-1.5B and 7B use 2048-token maximum responses, 1024-token maximum prompts, batches of 64 questions, and 16 rollouts per question.
  • Qwen3-8B: Qwen3-8B uses 4096-token maximum responses, 1024-token maximum prompts, batches of 512 questions, mini-batches of 32, and 16 gradient steps per batch.
  • Code models: Qwen2.5-coder-1.5B and 7B use 32768-token maximum responses, 2048-token maximum prompts, batches of 512, mini-batches of 32, and 16 rollouts per question.
  • Base models: Qwen2.5-base-1.5B and 7B use 4096-token maximum responses, 1024-token maximum prompts, batches of 512, mini-batches of 32, and 16 rollouts per question.
  • Dialogue templates: The appendix provides separate dialogue templates for math/CS tasks and code tasks.

A.4 An Example for ETS Top (20%) and GMTS (20%) with DAPO on Qwen3-8B under AIME2025

The example contrasts ETS Top (20%) with GMTS Top (20%), describing GMTS as more concise and accurate. The worked solution reduces the equation to two linear forms and counts the integer pairs satisfying their divisibility and range constraints.

  • GMTS Top (20%) is illustrated as producing an output that is more concise and accurate than ETS Top (20%).
  • Treating the equation as quadratic in x gives discriminant D = 289y2 and solutions x = 3y/4 or x = −2y/3.The quadratic-form setup uses a = 12, b = −y, and c = −6y2.
  • Inclusion-exclusion subtracts the 17 values of y divisible by 12, while the two branches overlap only at (0, 0).The zero solution is counted once, and the x-range check confirms both branches remain within [−100, 100].
  • The branch x = 3y/4 requires y divisible by 4, yielding 51 solutions for −100 ≤ y ≤ 100.Writing y = 4k gives −25 ≤ k ≤ 25.
  • The branch x = −2y/3 requires y divisible by 3, yielding 67 solutions over the same y range.Writing y = 3m gives −33 ≤ m ≤ 33.
Loading 2608.30632v1…