PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPO-$λ$: Credit Assignment improves LLM Reasoning

Prasanna Parthasarathi, Mathieu Reymond, Boxing Chen, Yufei Cui, Sarath Chandar

arXiv:2510.00194v1cs.LGcs.AI

TL;DR

GRPO’s critic-free return estimates limit fine-grained credit assignment across token sequences, while critic-based alternatives add memory and estimation challenges. GRPO-λ combines a policy-level reformulation of eligibility traces with GRPO group-return estimates and alternative token-weighting schemes. GRPO-λ improves 30-40% over GRPO on mathematical benchmark tasks and raises average performance by 3 points across five benchmarks, with over 4 points for the Deepseek-R1-Distill-Qwen-7B model.

  • Problem

    GRPO’s critic-free return estimates limit fine-grained credit assignment across token sequences, while critic-based alternatives add memory and estimation challenges.

    Source: §1 · p. 1, §2 · p. 3, §3 · p. 4

  • Method

    GRPO-λ combines a policy-level reformulation of eligibility traces with GRPO group-return estimates and alternative token-weighting schemes.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5

  • Results

    GRPO-λ improves 30-40% over GRPO on mathematical benchmark tasks and raises average performance by 3 points across five benchmarks, with over 4 points for the Deepseek-R1-Distill-Qwen-7B model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across the evaluated model sizes, architectures, training datasets, and mathematical benchmarks, critic-free eligibility traces provide a stronger alternative to GRPO.

    Source: §1 · p. 2, §6 · p. 9

    Training stability decreases, KL divergence is larger, the objective is sensitive, and experiments are limited to mathematical reasoning datasets.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed for tasks requiring complex reasoning, prompting significant interest in improving their reasoning abilities through post-training. Especially RL based methods using verifiable reward, like the state-of-the-art GRPO, have shown to tremendously improve reasoning behaviors when applied as post-training methods. However, the lack of an explicit reward or critic model limits GRPO's ability to assign fine-grained credit across token sequences. In this work, we present GRPO-$λ$, a novel extension to GRPO that enhances credit assignment in RL finetuning of LLMs for complex reasoning tasks. We approximate learning from $λ$-return with a reformulation of eligibility traces using token-level log-probabilities applied after each sequence generation, and a novel critic-free approximation of the temporal-difference error. We introduce a few variations for the weighting of the $λ$-return, and their applications to the eligibility-trace, where all the variations provide significant gains over GRPO. We compare GRPO-$λ$ against GRPO by training models from 1.5B to 7B parameters on $4$ different math reasoning datasets. The training plots demonstrate 30-40% improved performance during RL training on both LLaMA-3.1 and Qwen-2.5 architectures. Finally, we show that with GRPO-$λ$, the resulting average performance on AIME24, Math500, OlympiadMath, MinervaMath, and AMC improves over GRPO by over $3$ points and a $4.5$ points improvement on the 7B model.

1. Introduction

GRPO improves reasoning-oriented post-training but lacks fine-grained token credit assignment. GRPO-λ reformulates eligibility traces for GRPO and reports faster training plus higher benchmark performance.

Motivation

  • GRPO avoids a critic by averaging multiple policy rollouts, but consequently does not use eligibility traces to update earlier tokens.Eligibility traces balance Monte-Carlo variance against temporal-difference bias through λ.Source: §1 · pp. 1–2

Contribution

  • GRPO-λ applies reformulated eligibility traces directly to the policy, preserving GRPO’s lightweight memory footprint while propagating value information toward earlier tokens.The reformulation also motivates token-specific policy-gradient weighting and a critic-free approximation of value-estimation error.Source: §1 · p. 2

Results

  • Over 4 points average improvement is reported for R1-Distill-Qwen-7B across math benchmarks, including a 10-point gain on AIME24.Figure 1 summarizes the evaluation comparison.Source: Figure 1 · p. 2
  • 30-40% better learning performance than GRPO is reported across four mathematical reasoning tasks during RL training.The comparison covers different model sizes and architectures.Source: §1 · p. 2

2. Background

RL post-training treats token generation as sequential decision-making with rewards verified after completion. PPO uses a critic for value estimation, whereas GRPO removes that critic and estimates returns from grouped rollouts to reduce memory demands.

LLM reinforcement learning

  • In LLM post-training, states are generated-token sequences, actions are next-token choices, and an EOS action terminates the episode for answer verification.The prompt is the start state, and the pretrained LLM supplies the policy.Source: §2 · p. 3

PPO

  • PPO combines a policy with a value-function critic that bootstraps returns, trading lower variance for bias when the critic is inaccurate.Its clipped objective and generalized advantage estimation support policy updates.Source: §2 · p. 3

GRPO

  • GRPO removes the critic and approximates the expected return from multiple Monte-Carlo rollouts, avoiding the critic’s additional memory requirements.Its normalized advantage estimation replaces GAE in the policy objective.Source: §2 · pp. 3–4

PPO

  • Critic training is difficult because rewards are sparse and the pretrained policy can be substantially stronger than an initially mismatched value model.These issues motivate critic-free return estimation in GRPO.Source: §2 · p. 3

3. GRPO-𝜆for rapid reward propagation

GRPO-λ addresses GRPO’s biased token-level value estimates with a critic-free eligibility-trace reformulation and alternative weighting schemes. The resulting traces propagate reward information across tokens and improve performance over GRPO.

GRPO-λ motivation

  • GRPO’s group baseline estimates the expected return accurately only at the shared start state and is biased for later token states.Estimating each later state separately would require substantially more sequence generation, as in VinePPO.Source: §3 · p. 4

GRPO-λ method

  • GRPO-λ reformulates GAE traces as weighted cumulative action log-probabilities and substitutes GRPO’s normalized advantage for the unavailable critic-based TD error.λ=0 recovers GRPO, while computational overhead grows linearly with sequence length.Source: §3 · p. 4

GRPO-λ motivation

  • The value-estimation bias arises because later prefixes have fewer continuing sequences as EOS becomes more likely, making earlier states more accurately estimated.This analysis motivates position-dependent token weighting.Source: §3.1 · pp. 4–5

3.2. Alternative weighting mechanisms

  • Alternative weighting schemes adjust token importance by position, including direct trace weighting and a both variant that keeps early tokens as important as late tokens.The direct-weighting variant computes the trace once instead of repeatedly summing log-probabilities.Source: §3.2 · p. 5
  • All evaluated trace and weighting variations significantly improve over GRPO, indicating that per-token weighting can boost RL fine-tuning performance.Figure 2 depicts the classic recent traces and the proposed both traces.Source: §3.2 · p. 5, Figure 2 · p. 5

4. Experiments

Experiments evaluate GRPO-λ across model sizes, architectures, datasets, token-weighting schemes, training efficiency, stability, and benchmark performance. GRPO-λ generally improves training and evaluation over GRPO, while dataset choice and training stability remain important qualifications.

4.1. Analyzing the different token weighting schemes

  • Among token-weighting choices, both ε-weight and ε-trace have similar average performance, while λ=0.99 performs best across datasets and architectures.The experiments retain ε-trace because it is theoretically supported by GAE and aligns with PPO’s clipped surrogate objective.Source: §4.1 · p. 6
  • The both trace-weighting style systematically outperforms GRPO and sometimes exceeds the classic recent style, indicating that per-token weighting can improve RL fine-tuning.Figure 3 compares final accuracy rewards for both 1.5B models trained on ORZMath57K.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2. Training Efficiency

  • GRPO-λ improves training accuracy over GRPO across four models and multiple mathematical reasoning datasets, with average gaps of about 20–50%.The comparison excludes GSM8K in Figure 4 because some models include it in SFT or pretraining data.Source: §4.2 · pp. 7–8, Figure 4 · p. 7
  • GRPO-λ increases KL divergence relative to GRPO, and negative-advantage clamping can make the divergence orders of magnitude larger, reducing training stability.The experiments also report that explicitly incorporating the derived value-estimation bias into advantage estimation was detrimental to policy improvement.Source: §4.2 · p. 8

5. Limitations

GRPO-λ improves training and evaluation performance over GRPO but trades these gains for reduced training stability and uncertain generality beyond the studied mathematical reasoning setting.

  • GRPO-λ improves training and evaluation performance compared with GRPO, but its training stability decreases.The KL-divergence from the reference policy is larger for GRPO-λ and becomes orders of magnitude larger with negative-advantage clamping.Source: §5 · p. 9
  • The experiments cover mathematical reasoning datasets, so comparable gains on coding or general-purpose tasks remain unknown.Source: §5 · p. 9
  • Whether the improvement gap scales to substantially larger models, such as 32B parameters, remains unresolved.Source: §5 · p. 9
  • Explicitly incorporating the derived bias bound into advantage estimation was detrimental to policy improvement, indicating sensitivity in GRPO-λ’s objective.Source: §5 · p. 9

6. Conclusion and Future work

The paper concludes that GRPO-λ improves GRPO by rapidly propagating sequence rewards to relevant tokens through a generalized-advantage reformulation, while identifying trace design and stabilization as future directions.

6. Conclusion and Future work

  • GRPO-λ significantly outperforms GRPO across 4 training datasets, 5 mathematical benchmarks, 3 model sizes, and 2 model architectures.Source: §6 · p. 9
  • GRPO-λ reformulates generalized advantage estimation to rapidly back-propagate sequence rewards to relevant tokens without a critic.Source: §6 · p. 9
  • GRPO’s advantage term increasingly biases value estimates toward later tokens, motivating alternative weighting schemes that emphasize early tokens.Source: §6 · p. 9, §A.1 · p. 15
  • Alternative trace types can outperform classic GAE traces, motivating future analysis of domain- or data-dependent weighting schemes.Source: §6 · p. 9

A.2. Full Version of Theorem 1

The theorem and implementation reformulate GAE eligibility traces for critic-free GRPO, using token-level weighting and alternative trace schemes to improve credit assignment while retaining GRPO’s lightweight design.

A.2. Full Version of Theorem 1

  • GRPO-λ reparameterizes the policy-gradient estimate with a critic-free TD error and eligibility traces, extending GRPO’s group-return advantage estimation.Source: §A.2 · p. 16
  • The trace parameter λ∈[0, 1] preserves a bias-variance trade-off: λ=0 gives TD(0), while larger λ increases bias and reduces estimator variance.Source: §A.2 · p. 17
  • GRPO-λ includes ε-trace and ε-weight variants, with ε-trace applying traces before PPO clipping and ε-weight explicitly reweighting the loss.Source: §A.3 · p. 18
  • The method can apply trace weights directly to token-level policy losses, explicitly reweighting tokens according to their estimated temporal-difference contributions.Source: §A.3 · p. 18
  • The derived EOS-based upper bound for ΔV_t is not used in the implementation, and experiments adding it directly to the advantage were inconclusive.Source: §A.2 · p. 17, §A.3 · p. 18
  • The implementation uses precomputed lower-triangular trace matrices, and the both weighting style shows encouraging training and benchmark performance for exploring alternate schemes.Source: §A.3 · pp. 18–19

C. Training plots

Across the reported training plots, GRPO-λ achieves higher training accuracy than GRPO across λ values, weighting updates, and trace types, while maintaining a stable KL divergence.

  • GRPO-λ achieves higher training accuracy than GRPO regardless of λ, weighting update, or trace type.Source: C. Training plots · p. 21
  • KL divergence is higher for GRPO-λ than GRPO but remains stable throughout training.The reported stabilization depends on clamping negative advantages to −0.1.Source: C. Training plots · p. 21

D. Hyper-parameter comparisons

Additional training plots indicate that the observations from the main comparison remain valid across the other datasets.

E.1. Evaluation scores

This section reports benchmark results for Qwen2.5-Math-1.5B-Instruct, r1-Qwen-distill-1.5B, and R1-Distill-Llama-3B.

  • Benchmark results are reported for Qwen2.5-Math-1.5B-Instruct.Source: Table 3 · p. 27
  • Benchmark results are reported for r1-Qwen-distill-1.5B.Source: Table 4 · p. 28
  • Benchmark results are reported for R1-Distill-Llama-3B.Source: Table 5 · p. 29

E.2. Deepseek-R1-Distill-Qwen-7B

The 7B evaluation section reports benchmark results and generation-length comparisons, alongside hyperparameter comparisons spanning trace weights and update styles.

Related papers

Loading 2510.00194v1…