PaperlayerView exact revision on arXiv

Source-linked AI summary

GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

Jixiao Zhang, Chunsheng Zuo

arXiv:2504.09696v2cs.CL

TL;DR

GRPO-based mathematical reasoning is limited by sparse rewards, verbose outputs, and insufficient attention to problem difficulty. GRPO-LEAD addresses these issues with length-aware rewards, incorrect-answer penalties, and difficulty-aware advantage reweighting; evaluations report stronger accuracy, conciseness, and efficiency, including state-of-the-art performance for 14B-scale models.

  • Problem

    Existing GRPO implementations suffer from sparse reward signals, unnecessarily verbose reasoning, and inadequate encouragement of concise and precise solutions.

  • Method

    GRPO-LEAD extends GRPO with length-dependent accuracy rewards, explicit negative penalties for incorrect answers, and difficulty-aware advantage reweighting.

  • Results

    GRPO-LEAD evaluations on two AIME benchmarks report faster convergence, stronger reasoning capability, concise solution paths, and state-of-the-art performance for 14B-scale models.

  • Takeaways & Limitations

    The findings support combining reward shaping and difficulty modeling to develop more robust language-model reasoning for complex mathematical tasks.

  • Takeaways & Limitations

    Reported gains are specific to mathematical reasoning, and broader question-answering or logical-reasoning domains require further evaluation.

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO), which is widely adopted by R1-like reasoning models, has advanced mathematical reasoning. Nevertheless, GRPO faces challenges in reward sparsity, verbosity, and inadequate focus on problem difficulty. We propose GRPO-LEAD, enhancing GRPO with: (1) length-regularized rewards to encourage conciseness while maintaining accuracy; (2) explicit penalties for incorrect solutions to improve model precision; and (3) difficulty-aware advantage reweighting for robust generalization on challenging problems. Comprehensive evaluations demonstrate that GRPO-LEAD significantly improves reasoning accuracy, conciseness, and efficiency. Our approach achieves state-of-the-art performance for 14B-scale models, underscoring the synergy of our methods with appropriate model scale and high-quality data. Our source code, generated dataset, and models are available at https://github.com/aeroplanepaper/GRPO-LEAD.

1 Introduction

GRPO-LEAD targets reward sparsity, excessive verbosity, and weak difficulty sensitivity in GRPO-based mathematical reasoning. It combines length-regularized rewards, explicit incorrect-answer penalties, and difficulty-aware advantage reweighting, with evaluations reporting improved accuracy and conciseness.

  • Motivation: GRPO can produce weak learning gradients when responses within a question group are uniformly correct or incorrect.Binary accuracy rewards provide minimal differentiation in these cases, hampering convergence.
  • Motivation: Uniform reward signals also fail to encourage concise reasoning, increasing training and inference inefficiency.The paper identifies computational demands as a practical barrier, particularly in low-resource settings.
  • Contributions: Difficulty-based advantage reweighting scales question-level advantages according to empirical response correctness, prioritizing harder questions during policy updates.The framework uses this weighting to foster robust reasoning.
  • Contributions: GRPO-LEAD adds length-regularized rewards and explicit penalties for incorrect solutions to encourage concise reasoning while maintaining accuracy.The framework also applies difficulty-aware advantage reweighting to focus learning on harder problems.
  • Reported outcome: GRPO-LEAD evaluations report improved reasoning accuracy and conciseness, with state-of-the-art mathematical reasoning performance for 14B-scale models.The reported contribution is based on comprehensive evaluations.

2 Related Work

GRPO uses group-relative reward normalization and advantages to stabilize language-model fine-tuning, but fixed-budget training can encourage verbose reasoning. GRPO-LEAD replaces static length heuristics with rewards calibrated to relative response length and rollout accuracy.

  • GRPO: GRPO normalizes rewards within groups of responses to the same query and uses relative advantages for policy updates.The objective includes a clipping range for conservative updates.
  • GRPO: The importance sampling ratio compares the current policy’s token probability with the old policy’s probability for the same response context.This ratio appears in the GRPO objective formulation.
  • Limitations of existing GRPO: Large fixed reasoning budgets can enable reward hacking through extra reasoning and verification steps, producing verbose and less interpretable responses.The paper links this behavior to reduced reasoning efficiency and practicality.
  • Prior approaches: Prior length-reward methods use individual min-max normalization, cosine thresholds, or a predefined golden length.These approaches explicitly incentivize shorter or target-length responses.
  • GRPO-LEAD: GRPO-LEAD dynamically calibrates length-dependent rewards using each group’s relative response length and rollout accuracy rather than static length heuristics.Correct completions are rewarded in proportion to their relative conciseness.

3 Method

GRPO-LEAD modifies GRPO with length-dependent rewards, explicit penalties for incorrect responses, and difficulty-aware advantage reweighting. These mechanisms target verbosity, precision, and undertraining on challenging problems, while the experiments also examine SFT, data curricula, and targeted repetition penalties.

  • Core framework: GRPO-LEAD combines length-dependent accuracy rewards, explicit incorrect-response penalties, and difficulty-aware advantage reweighting.The method also examines how model scale and supervised fine-tuning affect RL fine-tuning.
  • Length-Dependent Accuracy Reward: Correct responses receive rewards based on relative conciseness, using standardized length deviation and exponential decay rather than static length thresholds.Overly long correct responses are penalized, while relatively concise responses are amplified according to each question’s response-length distribution.
  • Explicit Penalty for Incorrect Answers: Incorrect responses receive an explicit negative reward, making the expected reward positive only when P(correct) > 0.5 under negligible correct-answer length penalties.The authors report that this discourages speculative guessing and improves pass@1 and overall precision.
  • Advantage Reweighting for Difficulty-Aware Training: Difficulty-aware reweighting uses each question group’s empirical correctness ratio as an inverse difficulty proxy and applies a logistic factor to advantage estimates.Lower correctness ratios indicate harder problems, whose correct responses receive larger updates; incorrect responses on easier problems are penalized more strongly.
  • Training Strategy: SFT on 13k curated math problems followed by RL yielded faster convergence, while a two-stage hard-problem curriculum further enhanced the model’s ability.The curriculum used challenging DeepScaler problems and high-difficulty Light-R1 examples.
  • Targeted Reward Engineering: Removing length-dependent rewards and adding a −1.5 penalty for repeated n-grams improved precision and pass@1 by addressing repetitive output patterns.The intervention was used to mitigate a formatting issue associated with repetitive n-gram generation.

4 Experimental Setup

The experiments evaluate GRPO-LEAD across 7B and 14B DeepSeek-R1 Distilled Qwen variants using filtered and staged math datasets. They compare component additions and external baselines with Cons@32, Pass@1, and average output length under specified decoding and training settings.

  • Models and Framework: GRPO-LEAD is evaluated on DeepSeek-R1 Distilled Qwen variants at 7B and 14B parameter scales using the VERL framework.The integrated components are length-dependent accuracy rewards, explicit incorrect-solution penalties, and difficulty-aware advantage reweighting.
  • Training Data: The primary RL dataset contains approximately 9,000 DeepScaler questions after filtering out problems with difficulty ratings below 2.5.The 14B Stage 2 dataset further selects problems with stage-1 rollout accuracy no greater than 75% and adds Light-R1 numeric-answer problems, totaling 3,524 questions.
  • GRPO Configuration: GRPO uses 8 rollouts per question with group size 8, while the KL penalty term is removed because it suppressed exploration in the authors’ experiments.The cited setup passage begins with a batch-size fragment, but explicitly specifies group size and rollout count.
  • Hyperparameters: The length-reward decay uses α = 0.05, and difficulty reweighting uses A = 0.4, B = 1.5, ρ0 = 0.75, and k = 10.The configuration is intended to increase weighting sharply near the 75% correctness threshold while minimally reweighting easy problems.
  • 7B Experiments: The 7B study compares standard GRPO with sequential additions of length reward, advantage reweighting, and explicit penalties over approximately 200 training steps.The models are initialized from the DeepSeek-R1 Distilled 7B Qwen-Math checkpoint.
  • 14B Experiments: The 14B study applies SFT on 13k curated math problems before Stage 1 RL, then continues with a Stage 2 focus on underperforming problems and repetition-penalty training.The final 14B checkpoint is trained for 240 additional steps after Stage 2 begins, and test generation is limited to 14k tokens.
  • Metrics and Decoding: Evaluation reports Cons@32, Pass@1, and Average Length, using temperature 0.6 and 32 sampled solutions unless otherwise specified.Cons@32 measures majority-vote accuracy, Pass@1 measures top-1 correctness under a decoding strategy, and Average Length measures verbosity.

5 Results

Experiments on AIME24 and AIME25 show that GRPO-LEAD’s components accelerate training, improve accuracy, reduce verbosity, and perform strongly at 14B scale under a 14k-token budget.

  • Evaluation Setup: GRPO-LEAD evaluation covers training dynamics, component ablations, and comparisons with 14B-scale baselines on AIME24 and AIME25.The reported metrics include Cons@32, Pass@1, and average response length.
  • Training Dynamics: Adding length reward increases Pass@1 while accelerating early convergence by penalizing overly verbose correct solutions.The validation trajectory shows higher Pass@1 and faster early-stage convergence for GRPO+L than baseline GRPO.
  • Ablation Analysis: Length reward reduces average response length by 1,715 tokens on AIME24 and 1,903 tokens on AIME25 while improving performance.On AIME24, Pass@1 rises 1.6% (0.431 → 0.438); on AIME25, it rises 5.4% (0.292 → 0.308).
  • Ablation Analysis: Difficulty-aware advantage reweighting further improves Pass@1, increasing it from 0.438 to 0.458 on AIME24 and from 0.308 to 0.325 on AIME25.On AIME25, Cons@32 also increases from 0.533 to 0.567, while AIME24 Cons@32 remains 0.767.
  • Ablation Analysis: The explicit incorrect-solution penalty yields the highest Pass@1, reaching 0.470 on AIME24 and 0.345 on AIME25.On AIME24, Cons@32 rises to 0.800, while average solution length increases modestly but remains shorter than the Deepseek-7B baseline.
  • Comparison with Baselines: At 14B scale under a 14k-token budget, LEAD-stage2 achieves the best reported Pass@1 and Cons@32 while reducing average solution length to 8,267 tokens on AIME24 and 8,668 on AIME25.LEAD-stage1 already matches Light-R1-14B-DS on Cons@32 and outperforms DeepSeek-14B on both benchmarks’ reported comparisons.

6 Conclusion

GRPO-LEAD extends GRPO with length-dependent rewards, explicit negative penalties, and difficulty-aware advantage reweighting for mathematical reasoning. Evaluations report faster convergence, stronger reasoning, concise solution paths, and state-of-the-art performance at 14B scale.

  • GRPO-LEAD adds length-dependent accuracy rewards, explicit negative penalties, and difficulty-aware advantage reweighting to address challenges in structured problem-solving.The components discourage verbosity, clarify correct-versus-incorrect reward boundaries, and prioritize tougher problems.
  • Empirical evaluations on two AIME benchmarks show faster convergence, stronger reasoning capability, and concise solution paths.
  • GRPO-LEAD achieves state-of-the-art mathematical reasoning performance in 14B-scale experiments while balancing output brevity with high problem-solving accuracy.

7 Limitations

The reported limitations concern both evaluation scope and experimental resources. Results are specific to mathematical reasoning, while limited compute prevented comprehensive experiments and formal hyperparameter search.

  • The reported gains are specific to mathematical reasoning, and effectiveness on broader question-answering or logical reasoning domains remains to be evaluated.These domains may differ in correctness signals and structural characteristics.
  • The authors could not formally perform a hyperparameter search to justify the choices for their designed modifications.

A Evaluations on Coding Tasks

On LiveCodeBench coding tasks, LEAD-14B slightly outperforms the original DeepSeek-R1-Distill-Qwen-14B baseline in accuracy but produces somewhat longer completions. The result suggests math-trained compression does not transfer equally to code generation.

  • LEAD-14B is evaluated against DeepSeek-R1-Distill-Qwen-14B on LiveCodeBench release_v5, comprising 880 code-generation tasks with an 8k-token maximum sequence length.Results are summarized in Table 3.
  • 0.5156 vs. 0.5103 accuracy: LEAD-14B outperforms the DeepSeek-R1-Distill-Qwen-14B baseline while producing slightly longer completions.
  • The authors hypothesize that longer coding completions reflect training exclusively on mathematical reasoning datasets, where the method compresses reasoning paths more effectively.

B Detailed Analysis on AIME25 by Difficulty

Difficulty-stratified AIME25 results indicate that advantage reweighting provides larger benefits on difficult problems, while explicit penalties improve precision on the most difficult set. The combined method achieves the strongest reported Cons@32 accuracy there.

  • Difficulty stratification: AIME25 problems are stratified as normal (1–5), difficult (6–10), and highly difficult (11–15) based on problem number.Table 4 reports evaluation results for each difficulty stratum.
  • Advantage reweighting: 1.95% improvement on normal problems versus 13.7% on difficult problems: GRPO+LAD’s Pass@1 advantage over GRPO+L widens with difficulty.
  • Advantage reweighting: GRPO+L and GRPO+LAD have identical Pass@1 on highly difficult problems, while neither method’s lack of explicit penalties prevents numerous wrong solutions.This produces unstable majority-voting accuracy, especially for intrinsically difficult problems.
  • Explicit penalty: 0.4 Cons@32: GRPO+LEAD achieves the highest accuracy on highly difficult problems and more than doubles GRPO+L (0.172) and GRPO+LAD (0.156) precision.It generates comparable numbers of correct answers despite producing many fewer total answers.

C Qualitative Analysis of Solution Conciseness

A qualitative comparison shows that GRPO+L produces shorter, more focused correct solutions than Deepseek-7B, avoiding repetition and conversational digressions.

  • C Qualitative Analysis of Solution Conciseness: GRPO+L produces a tight, step-by-step correct solution that avoids repetition and concludes efficiently.The comparison concerns the shortest correct rollouts for AIME 25 I, Problem 3.
  • C Qualitative Analysis of Solution Conciseness: Deepseek-7B uses repeated self-checks and conversational digressions, nearly doubling the solution length and reducing clarity.Italicized text identifies meta-commentary or self-correction loops in the Deepseek-7B rollout.
  • C Qualitative Analysis of Solution Conciseness: The case study indicates that the length-reward mechanism encourages a more disciplined and economical reasoning style.
Loading 2504.09696v2…