PaperlayerView exact revision on arXiv

Source-linked AI summary

GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy

Hongze Tan, Zihan Wang, Jianfei Pan, Jinghao Lin, Hao Wang, Yifan Wu, Tao Chen, Zhihang Zheng, Zhihao Tang, Haihua Yang

arXiv:2508.04349v6cs.CLcs.AI

TL;DR

GRPO-style reinforcement learning uses coarse-grained rewards that assign the same outcome-based signal across tokens, limiting credit assignment in complex reasoning. The paper introduces Dynamic Entropy Weighting through GTPO for token-level rewards and GRPO-S for sequence-level rewards. Experiments report consistent superiority over DAPO and GRPO baselines, while the authors identify relative-entropy comparison as an open question.

  • Problem

    GRPO assigns a uniform final-outcome reward to every token, making credit assignment coarse in long reasoning sequences.

  • Method

    Dynamic Entropy Weighting uses policy entropy to redistribute rewards through GTPO’s token-level supervision and GRPO-S’s average-entropy sequence-level modulation.

  • Results

    The methods consistently outperform strong DAPO and GRPO baselines across multiple reasoning benchmarks.

  • Takeaways & Limitations

    Entropy can serve as an active reward-shaping signal for more fine-grained credit assignment without an explicit value network.

  • Takeaways & Limitations

    The optimal method for measuring relative entropy remains an open question, with column-wise, row-wise, and matrix-wise comparisons identified as alternatives.

Abstract

from arXiv · show

Reinforcement Learning (RL) is pivotal for enhancing Large Language Model (LLM) reasoning, yet mainstream algorithms such as GRPO and DAPO remain constrained by a coarse-grained credit assignment paradigm, where all tokens within the same response receive the identical reward. In this paper, we propose Dynamic Entropy Weighting, systematically define entropy-based weight ratios $\frac{H_{i,t}}{\sum_{k=1}^{n} H_{k,t}}$ and similar variants to redistribute rewards and get fine-grained rewards through two new algorithms: Group Token Policy Optimization (GTPO), which assigns an entropy-weighted reward to each token and synthesizes token-specific advantage function to drive the model toward optimal path, and the analogous algorithm Sequence-Level GRPO (GRPO-S), which extends this design to the sequence level and exhibits superior stability in long Chain-of-Thought (CoT) reasoning tasks.

1. Introduction

The paper addresses coarse-grained credit assignment in GRPO-style reinforcement learning by using policy entropy to redistribute rewards at token and sequence levels. It introduces GTPO and GRPO-S, supported by theoretical analysis and experiments reporting stronger performance than DAPO and GRPO baselines.

  • Motivation: GRPO assigns every token the same reward from the final outcome, so long reasoning chains can penalize correct steps alongside one error.The paper identifies this as a central limitation for complex and long Chain-of-Thought reasoning.
  • Dynamic Entropy Weighting: Dynamic Entropy Weighting redistributes rewards toward critical decision points instead of treating policy entropy only as a regularizer or selection mask.It reinforces high-entropy exploration in correct solutions and penalizes confident errors in incorrect ones without an explicit value network.
  • GTPO: GTPO assigns unique entropy-weighted rewards to individual tokens, enabling fine-grained credit assignment within the efficient GRPO framework.The method rewards high-entropy tokens in successful sequences while modulating tokens in unsuccessful sequences through the framework’s dual strategy.
  • GRPO-S: GRPO-S applies entropy weighting at the sequence level, modulating an entire sequence’s reward by its average entropy for greater sequence-level stability.It complements GTPO’s token-level precision with a lightweight macro-level variant.
  • Evaluation: The paper provides theoretical analysis of unbiasedness and convergence and reports that its methods significantly outperform strong baselines on challenging reasoning benchmarks.The conclusion specifically describes consistent improvements over DAPO and GRPO across multiple reasoning benchmarks.

2. Dynamic Entropy Weighting For Policy Optimization

The framework addresses GRPO’s coarse-grained credit assignment by reshaping rewards with dynamic entropy weighting, producing token- or sequence-sensitive learning signals. GTPO redistributes rewards across tokens, while the analysis shows reward-mass conservation, asymptotic consistency, and shared global optima with established objectives.

  • Motivation: GRPO applies one sequence-level advantage uniformly to every token, creating coarse-grained credit assignment and response-length dependence.Token-level baselines reduce variance relative to sequence-level means, while uniform rewards fail to capture token specificity.
  • Dynamic Entropy Weighting: Dynamic entropy weighting focuses policy gradients on critical decision points by reshaping uniform rewards into fine-grained signals.The framework separates successful and unsuccessful sequences, rewarding high-entropy tokens in successful paths and penalizing low-entropy tokens in unsuccessful paths more heavily.
  • GTPO: GTPO assigns entropy-weighted rewards at the token level, normalizing positive rewards across successful sequences and using inverse entropy to penalize confident mistakes.The resulting token rewards are converted into separate positive and negative advantage functions and optimized with a token-level PPO-style objective.
  • Theoretical Guarantees: When α1 + α2 = 1, GTPO conserves the total shaped positive reward at each timestep, preserving the original binary reward mass.This conservation underpins the claim that GTPO is an unbiased augmentation of the DAPO objective in total learning signal.
  • Theoretical Guarantees: The analysis states that GTPO is asymptotically consistent and shares the same global optimum as DAPO.The entropy term is detached from gradient computation, so shaping changes training dynamics while preserving the optimization target under the stated conditions.

3. Experiments

Experiments evaluate GTPO and GRPO-S on long-horizon mathematical reasoning benchmarks across Qwen2.5 model sizes, comparing them with GRPO and DAPO baselines. Both proposed methods consistently outperform the baselines, converge rapidly, and improve exploration-sensitive performance.

  • 3.2. Comparative Performance Analysis: GTPO and GRPO-S consistently outperform GRPO, DAPO, and DAPO w/ Forking Tokens across all tested configurations, including MATH 500.The comparison covers AIME 2024, AIME 2025, and MATH 500 on Qwen2.5-7B/32B models.
  • 3.2. Comparative Performance Analysis: GTPO achieves a +29.4-point APG on AIME 2024 Pass@32 for Qwen2.5-7B, compared with +9.9 points for Qwen2.5-32B.The larger gain on the smaller model indicates stronger benefits from entropy weighting in that setting.
  • 3.3. Reward Trajectories and Sample Efficiency: Mean reward trajectories show higher final reward ceilings for GTPO and GRPO-S with convergence largely within 210 training steps.The reported convergence speed indicates that improved exploration did not require slower learning.
  • 3.4. Hyperparameter Analysis: Sensitivity analysis finds robust performance gains over DAPO on Mean@32 and Pass@32 across tested reward-shaping settings, with GRPO-S showing higher stability.The hyperparameter comparison examines the trade-off between exploration and exploitation.

4. Discussion

The discussion frames batch-level entropy comparison as an implicit curriculum and identifies relative-entropy comparison as an unresolved design choice. Learning priority shifts automatically as the model becomes proficient on previously difficult problems.

  • Batch-Level Entropy Comparison as Implicit Curriculum Learning: Batch-level entropy comparison directs larger gradients toward solvable, high-entropy problems near the model’s capability frontier.As proficiency increases and entropy falls, the learning focus shifts to the next challenging tasks without manually designed curricula.
  • Future Work: The optimal way to compare relative entropy remains open among column-wise, row-wise, and matrix-wise alternatives.These alternatives compare same-position tokens across responses, tokens within one response, or all tokens collectively.

5. Conclusion

The paper addresses coarse-grained credit assignment in LLM reasoning with dynamic entropy weighting, implemented through GTPO and GRPO-S. Across multiple benchmarks, the methods outperform GRPO and DAPO baselines while using policy uncertainty to concentrate learning on critical decisions.

  • 5. Conclusion: Dynamic entropy weighting introduces GTPO for token-level supervision and GRPO-S as a computationally efficient sequence-level variant.GTPO assigns fine-grained token rewards, while GRPO-S applies entropy weighting at the sequence level.
  • 5. Conclusion: The methods consistently outperform strong GRPO and DAPO baselines across multiple reasoning benchmarks.The conclusion attributes this outcome to the proposed entropy-weighting mechanism.
  • A.2. The Rise of Value-Function-Free Policy Optimization: GRPO uses group-average rewards without a value function, reducing memory and computational costs but remaining sensitive to reward noise and training instability.This value-function-free design motivates the paper’s development of more refined optimization methods.
  • A.3. A Technical Comparison with the DAPO Baseline: DAPO improves GRPO-style training with techniques including Clip-Higher and Dynamic Sampling, but its token-level advantage remains constant within each sequence.The paper distinguishes DAPO’s loss normalization from GTPO’s direct reconstruction of the reward signal.
  • 5. Conclusion: The variance analysis assumes i.i.d. token rewards to derive properties of the global mean estimator.This assumption simplifies formal analysis despite autoregressive token generation.

B.1. Variance Comparison of Two Mean Calculation Methods

The variance comparison shows that directly averaging all token-level samples is no worse, and generally better for unequal sequence lengths, than averaging subgroup means. Applied to RL training, this supports token-level mean estimation over GRPO’s sequence-level two-stage averaging.

  • Variance Comparison: Directly computing the grand mean has smaller or equal variance than first averaging subgroup means, with strict improvement when group sizes differ.The result follows from the arithmetic-mean/harmonic-mean comparison under the stated sampling assumptions.
  • Variance Comparison: In RL training, the two estimators correspond to averaging across all tokens versus averaging sequence means, where unequal response lengths yield lower variance for the token-level estimator.The analysis maps m to the number of sequences and n_i to each sequence length.
  • Conclusion: The analysis concludes that a single average over all tokens should replace GRPO’s two-stage averaging when GRPO is unified at the token level.The corresponding variance inequality is stated as Var(R_hat2) ≤ Var(R_hat1).
  • Application to GRPO: GRPO and DAPO assign a sequence’s final reward to every token, so their reward estimation effectively uses sequence-level means despite token-level optimization.This treatment ignores response length when constructing the token-level reward signal.
  • Application to GRPO: Using the token-level mean reduces estimator variance and supports more stable gradients, but exposes GRPO’s length dependency and coarse credit assignment.For correct responses, the sequence-level design can implicitly favor longer outputs because the advantage does not capture token specificity.

B.3. Theoretical Analysis of Reward Conservation and Convergence

GTPO’s entropy-weighted shaping conserves the total positive reward mass of GRPO and, under entropy consolidation, its gradient bias vanishes asymptotically, recovering the standard GRPO objective.

  • Conclusion: GTPO encourages exploration while entropy variance is high, then asymptotically reduces to the standard GRPO objective.Thus, the shaping changes the transient learning signal while preserving valid convergence toward the baseline objective.
  • Reward Conservation: GTPO preserves the sum of original positive rewards at every timestep when α1 + α2 = 1.The shaped positive rewards therefore retain the original reward mass used for advantage calculation.
  • Convergence Assumption: The proof assumes that entropy variation among successful sequences diminishes as training progresses, with regularization preventing singular ratios.Under this entropy-consolidation condition, individual entropy ratios approach unity.
  • Gradient Convergence: As entropy ratios converge to one, the reward deviation from the original reward vanishes for successful sequences.The resulting gradient difference is controlled through the score function and bounded-gradient assumptions.

B.4. Analysis of GRPO-S

GRPO-S applies sequence-level entropy weighting while preserving positive reward mass and achieving asymptotic gradient consistency under boundedness and entropy-consolidation assumptions.

  • Sequence-Level Reward Conservation: GRPO-S preserves the sum of original positive sequence rewards when β1 + β2 = 1.The conservation result holds at the group level for successful sequences.
  • Gradient Consistency: With bounded score functions and bounded clipped rewards, the expected GRPO-S gradient converges to the GRPO gradient in L1 norm.The proof applies dominated convergence after establishing advantage convergence.
  • Reward and Advantage Convergence: Under entropy consolidation, GRPO-S shaped rewards converge pointwise to the original rewards, bringing their means, standard deviations, and advantages into agreement.The argument uses continuity of the advantage function when the reward standard deviation is nonzero.
  • Global Optimum: The analysis states that GRPO-S shares the same global optimum as DAPO, while exact β1 + β2 = 1 is treated as unnecessary because it only rescales the objective.The experimental setting keeps the coefficient sum close to one without enforcing strict equality.
  • Implementation Variant: GRPO-S also admits a geometric-mean sequence importance weight as a lower-variance proxy, although this introduces bias for long-sequence stabilization.This is presented as an alternative to the arithmetic-mean importance weight.

E.1. Analysis of Training Dynamics: Entropy Rebound and Exploration

GTPO and GRPO-S produce an entropy rebound after an initial decline, unlike DAPO’s continued entropy collapse, indicating sustained exploration during training.

  • Exploration: The rebound is presented as evidence that entropy-weighted reward shaping incentivizes the model to maintain exploration.The text contrasts this with DAPO’s premature convergence and policy collapse.

E.2. Analysis of Generation Characteristics: Response Length and Clipping

GTPO and GRPO-S generate longer, more exploratory responses and achieve a higher final reward ceiling than DAPO across the reported Pass@k trajectories.

  • Response Length Clipping: GTPO and GRPO-S have a higher response-length clip ratio, with roughly 10% of sequences reaching the 4096-token limit compared with DAPO.The higher ratio is interpreted as evidence of more elaborate reasoning that often uses the full generation budget.
  • Reward Trajectories: GTPO and GRPO-S consistently reach a higher final reward ceiling than DAPO across the reported Pass@k metrics.Figures 8 and 9 are described as consistent with the main-paper trends for Pass@4 and Pass@16.

E.4. Reward Trajectories on Training Sets

GTPO and GRPO-S achieve substantial performance improvements without reduced sample efficiency: training reward trajectories for all models largely converge by approximately 210 steps.

  • By approximately 210 training steps, all models largely converge on mean reward, including GTPO, GRPO-S, and the baseline.
  • The comparable convergence timing indicates that the methods’ performance improvements do not result from longer training.
  • The results attribute the gains to a more effective learning signal derived from entropy weighting.

F. Qualitative Case Study

The qualitative case study contrasts successful GTPO and GRPO-S reasoning with DAPO’s failure on a long-chain mathematical problem, illustrating the practical difference between entropy-aware and coarse-grained credit assignment.

  • GRPO-S solution: The GRPO-S solution counts sets by summing 2^(a_i-1) over elements of A, then uses the unique binary representation of 2024 to identify seven exponents.
  • Qualitative comparison: GTPO and GRPO-S obtain the correct sum of 55, whereas DAPO produces the incorrect result 64.The GTPO and GRPO-S solutions derive n = 7 and sum the elements of A to 55; DAPO selects ten elements and obtains 64.
  • DAPO solution: The DAPO solution uses the same counting setup but selects A = {1, 3, 4, 5, 6, 7, 8, 9, 10, 11}, whose elements sum to 64.
  • DAPO failure: DAPO learns the solution template but makes a critical binary-decomposition error, yielding an overconfident but incorrect reasoning path.
  • DAPO failure: The case study attributes DAPO’s failure to a coarse sequence-level reward that cannot penalize subtle intermediate mistakes.The analysis characterizes DAPO as receiving a single +1 or -1 for the entire sequence.
  • Entropy-aware methods: GTPO and GRPO-S use entropy-aware reward signals designed to penalize low-entropy, high-confidence mistakes and reward exploration in successful paths.
Loading 2508.04349v6…