PaperlayerView exact revision on arXiv

Source-linked AI summary

GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning

Ziru Liu, Cheng Gong, Xinyu Fu, Yaofang Liu, Ran Chen, Shoubo Hu, Suiyun Zhang, Rui Liu, Qingfu Zhang, Dandan Tu

arXiv:2507.10628v2cs.LGcs.AI

TL;DR

On-policy RLVR faces sparse rewards when training difficulty exceeds a model’s capabilities. GHPO adapts guidance to sample difficulty, switching between guided imitation and on-policy reinforcement learning. It reports an average performance gain of approximately 5% across six mathematics benchmarks and outperforms state-of-the-art RL methods.

  • Problem

    Capacity-difficulty mismatch can produce sparse rewards; in GRPO, zero advantage leaves queries without a learning signal and wastes generation and evaluation compute.

    Source: §1 · p. 2, §2.3 · p. 4

  • Method

    GHPO detects sample difficulty and adaptively switches between on-policy reinforcement learning and guided imitation learning using prompt refinement.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Approximately 5% average performance gain across six challenging mathematics benchmarks; GHPO outperforms state-of-the-art RL methods with consistent gains across model families.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    GHPO targets improved training stability and efficiency by addressing capacity alignment and reward sparsity in RLVR.

    Source: §1 · p. 2

    Static guidance cannot adapt as model capabilities evolve, risking over-guidance on problems the model could solve through exploration and reduced sample efficiency.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for facilitating the self-improvement of large language models (LLMs), particularly in the domain of complex reasoning tasks. However, prevailing on-policy RL methods often contend with significant training instability and inefficiency. This is primarily due to a capacity-difficulty mismatch, where the complexity of training data frequently outpaces the model's current capabilities, leading to critically sparse reward signals and stalled learning progress. This challenge is particularly acute for smaller, more resource-efficient LLMs. To overcome this, we introduce the Guided Hybrid Policy Optimization (GHPO), a novel difficulty-aware reinforcement learning framework. GHPO dynamically calibrates task difficulty by employing adaptive prompt refinement to provide targeted guidance. This unique approach adaptively balances direct imitation learning for problems currently beyond the model's reach with exploration-based reinforcement learning for more manageable tasks, effectively creating a smooth and optimized learning curriculum. Extensive experiments demonstrate that GHPO achieves an average performance gain of approximately 5% across six challenging mathematics benchmarks, consistently outperforming strong on-policy reinforcement learning and curriculum learning baselines. Further analysis confirms that our framework significantly enhances both training stability and final reasoning performance, thus offering a scalable and efficient solution for developing powerful and robust reasoning models.

1 Introduction

The introduction frames sparse rewards and inefficient on-policy learning as consequences of a mismatch between training-task difficulty and model capacity. GHPO addresses this with adaptive guidance that combines on-policy exploration and imitation learning; experiments report gains across six mathematics benchmarks.

1 Introduction

  • On-policy RLVR can suffer sparse rewards when training-data difficulty exceeds model capacity, while increasingly challenging data can reduce sample efficiency, especially for compact models.Related approaches include manual curriculum partitioning, dynamic sampling that discards prompts, and hybrid methods requiring a resource-intensive auxiliary LLM.Source: §1 · p. 2
  • GHPO detects sample difficulty and adaptively switches between on-policy reinforcement learning and guided imitation learning through prompt refinement.It uses partial ground-truth solution traces for difficult samples while preserving exploration on manageable ones.Source: §1 · p. 2
  • GHPO outperforms state-of-the-art RL methods with consistent gains across six mathematics benchmarks and different model families.The experiments are presented as validating the approach’s effectiveness and robustness.Source: §1 · p. 2

2 Preliminaries

The preliminaries introduce RLVR and GRPO, then explain how reward sparsity can leave difficult queries without a learning signal. Such queries waste computation and can make effective batch size fluctuate, destabilizing training.

2.1 Reinforcement Learning with Verifiable Rewards (RLVR)

  • RLVR fine-tunes an LLM policy to generate token trajectories for prompts, with a verifier assigning sparse terminal rewards based on answer correctness.The training objective maximizes expected terminal reward over the prompt distribution.Source: §2.1 · p. 3

2.2 Group Relative Policy Optimization (GRPO)

  • GRPO estimates a response’s advantage by normalizing its reward against the reward statistics of responses sampled for the same prompt.The response-level advantage is shared across all token-generation steps in that response.Source: §2.2 · p. 3
  • GRPO increases the log-probability of tokens in high-advantage trajectories and may use clipping and a KL-divergence penalty to regularize policy updates.Some recent implementations omit the KL term.Source: §2.2 · p. 3

2.3 The Challenge of Reward Sparsity in GRPO

  • A query yielding only incorrect responses has zero rewards and zero advantages, so its policy gradient vanishes and the sampled responses provide no learning signal.When many difficult queries occur in a batch, much of the generation and evaluation computation is wasted.Source: §2.3 · p. 4
  • Fluctuations in the number of queries with non-zero learning signals introduce noise into gradient estimates and can destabilize training and impede reliable convergence.Source: §2.3 · p. 4
  • Qwen2.5-7B-Instruct failed to solve 52% of approximately 900,000 competition-level problems in NuminaMath-1.5, illustrating the capacity-difficulty mismatch for smaller models.The passage notes that this model is more capable than its Qwen2.5-7B-Base counterpart.Source: §2.3 · p. 4

3 The Proposed Framework

GHPO addresses the difficulty of aligning guidance with a model’s changing capabilities by dynamically switching between on-policy reinforcement learning and guided learning. Its framework detects difficult queries and progressively refines their prompts with ground-truth solution hints.

3.1 From Static Guidance to a Dynamic Framework

  • For difficult queries, GHPO uses partial ground-truth solution traces to provide learning signals where sparse rewards might otherwise leave none.The framework’s motivating assumption is that trace guidance can improve out-of-distribution generalization on problems the base policy fails.Source: §3.1 · pp. 4–5
  • Static guidance requires subjective, laborious manual curation and can over-guide problems that become solvable through exploration, reducing sample efficiency.A problem’s difficulty can change as the policy improves.Source: §3.1 · p. 5

3.2 Guided Hybrid Policy Optimization (GHPO)

  • GHPO dynamically assesses sample difficulty and switches between on-policy reinforcement learning and guided learning, reserving guidance for problems beyond the model’s current capabilities.It combines automated difficulty detection with adaptive prompt refinement.Source: §3.2 · p. 5

3.3 Automated Difficulty Detection

  • Difficulty detection analyzes rewards from multiple sampled responses and flags a query as difficult when all sampled rewards are zero.The method uses accuracy rewards already present in training rather than additional manual effort.Source: §3.3 · p. 6

3.4 Adaptive Prompt Refinement with Multi-Stage Guidance

  • Adaptive prompt refinement starts with a small hint and increases its length after incorrect responses, using hint ratios of 0.25, 0.5, and 0.75.The schedule retains the original query for advantage computation when at least one response is correct, and increases guidance when none is correct.Source: §3.4 · p. 7

3.5 Cold-Start Strategy

  • An optional cold start runs ordinary GRPO for the first 20 optimization steps before enabling difficulty detection.This is intended to build formatting capability while conserving early computational resources and avoiding initial detection bias.Source: §3.5 · p. 7

3.6 Hint Extraction Example

  • In the triangle example, GHPO appends 50% of the ground-truth solution’s characters as hints after repeated incorrect responses.The example illustrates prompt refinement for a difficult query with no valid reward signal.Source: §3.6 · p. 7, Figure 4 · p. 8

4 Experiment

The experiments evaluate GHPO on verifiable mathematical tasks using training data of two difficulty levels. The setup includes two Qwen base models and GRPO, curriculum-learning, and constant-hint comparison models.

4 Experiment

  • The evaluation demonstrates GHPO’s efficacy on verifiable mathematical tasks, although the method is designed for general applicability.The experiments use two training datasets constructed from MATH and NuminaMath-1.5.Source: §4.1 · p. 8, §4 · p. 8

4.1 Training Datasets

  • Math3to5 contains 8,890 Level 3–5 competition problems, while NuminaMath-S contains 18,300 more challenging problems with step-by-step solutions.NuminaMath-S combines math3to5 with additional OlympiadBench and AMC problems.Source: §4.1 · p. 8

4.2 Baseline Models

  • The comparisons include Qwen2.5-Base-7B and Qwen2.5-Math-7B models, GRPO-trained variants, a two-stage curriculum model, and a constant-hint curriculum variant.The curriculum model moves from math3to5 training to more challenging OlympiadBench and AMC problems.Source: §4.2 · p. 8

4.3 Implementation Details

Training used rule-based accuracy and format rewards, while evaluation used standard math-reasoning benchmarks and competition-level tasks. The section also identifies performance tables for models trained on Math and Mixed datasets.

4.3 Implementation Details

  • Training used rule-based answer rewards and format rewards at a 2:1 weight ratio, with AdamW, a cosine-decayed learning rate, and no KL penalties.The answer reward was +1 for correct answers and 0 otherwise; the format reward was +1 for the specified reasoning-and-answer format. Training used batch size 112 and eight sampled responses per query.Source: §4.3 · p. 9

4.4 Evaluation Details

  • Evaluation reused the training prompt template without hints and used generation limits of 4096 tokens, reporting pass@1 for most benchmarks and avg@32 for AIME2024.Benchmarks included MATH_500, OlympiadBench, Minerva Math, GPQA-Diamond, AMC2023, and AIME2024.Source: §4.4 · p. 9
  • The section includes a performance-comparison table for models trained on the Math dataset and another for models trained on the Mixed dataset.The supplied captions do not specify the tables’ row or column layout.Source: Table 1 · p. 9, Table 2 · p. 9

4.5 Overall Performance

GHPO outperforms conventional methods across datasets of varying difficulty, with gains reported against GRPO and curriculum-learning baselines. The section also examines how curriculum learning and fixed hints compare with GHPO.

Performance on Medium-Difficulty Tasks

  • GHPO achieves an average accuracy improvement of 4.4% over GRPO on the math3to5 training dataset, with gains exceeding 8% on AMC2023 and GPQA-Diamond.The passage attributes the improvement primarily to mitigating reward sparsity.Source: §4.5 · p. 10

Performance on Challenging Tasks

  • GHPO improves accuracy across five of six benchmarks over both GRPO and curriculum learning on challenging NuminaMath-S tasks, including AIME2024 gains from 0.122 to 0.163 over GRPO.The comparison uses the identical training dataset for GHPO and GRPO.Source: §4.5 · p. 10

Impact curriculum learning and Fixed Hints

  • Curriculum learning consistently falls short of GHPO; adding fixed 50% ground-truth traces raises its reported result from 0.415 to 0.422, still below GHPO’s 0.442.The authors note that reliable fine-grained dataset difficulty partitioning can be challenging in practice.Source: §4.5 · p. 10

Performance evaluation

  • Figure 5 depicts the proportion of problems detected as difficult within a mini-batch.Source: Figure 5 · p. 10

4.6 Training Dynamics

GHPO’s training dynamics combine higher accuracy reward and smaller gradient magnitudes than GRPO, alongside longer later-stage responses. The paper also reports that difficult problems remain prevalent during training.

Training dynamics

  • Approximately 60% of problems continued to exceed the model’s capabilities in later training stages, underscoring persistent reward sparsity.Figure 5 tracks the proportion of problems requiring hints within a mini-batch.Source: §4.6 · pp. 10–11
  • GHPO exhibits a clear accuracy-reward advantage over GRPO throughout training, while both methods reach comparable near-maximal format reward early and sustain it.Source: §4.6 · p. 11, Figure 6 · p. 11
  • GHPO generated significantly longer responses than GRPO in later training, while both methods’ mean response lengths increased over time.The authors suggest exposure to partial ground-truth solutions may guide longer reasoning paths.Source: §4.6 · p. 11, Figure 6 · p. 11
  • GHPO maintains smaller gradient magnitudes than GRPO, which the authors interpret as indicating smoother, more stable optimization.The authors describe the overall trajectory as more controlled policy updates.Source: §4.6 · p. 11, Figure 6 · p. 11

4.7 Generalization to other models

GHPO also improves performance when applied to the more capable, math-specialized Qwen2.5-Math-7B base model. The authors present this as evidence of benefits across different foundational model strengths.

Generalization to other models

  • GHPO consistently outperforms GRPO when both are applied to the Qwen2.5-Math-7B base model.The authors describe this model as a stronger, math-specialized foundation than the general-purpose Qwen2.5-Base-7B used earlier.Source: §4.7 · p. 12

4.8 Case Study

The case study contrasts GRPO’s repeated failure on a challenging problem with GHPO’s use of a hint-guided prompt. With the hint, GHPO training produces at least one correct reasoning path.

4.8 Case Study

  • The case study compares GRPO and GHPO on a particularly challenging mathematical problem, with its ground-truth solution shown alongside it.Table 3 presents the problem and ground-truth solution; Table 4 shows the extracted hint and constructed GHPO prompt.Source: §4.8 · p. 12
  • GHPO constructs the hint-guided prompt by concatenating the first 50% of the ground-truth solution to the original query.The case study presents this as targeted guidance for the challenging problem.Source: §4.8 · p. 12
  • GHPO training produces at least one correct reasoning path with hint guidance, while standard GRPO often diverges from the ground-truth solution.The GHPO response follows the hint and reaches a correct solution; the GRPO example is incorrect.Source: §4.8 · p. 12

5 Related Work

Related work spans zero-RL training refinements, approaches to balancing imitation and exploration, and methods for addressing sparse rewards. GHPO is positioned as using adaptive difficulty detection and prompt refinement to make use of all training data.

5 Related Work

  • DeepSeek-R1 [2] exemplifies zero-RL training with rule-based rewards, while SimpleRL-Zoo studies this training across base models and sizes.The cited work aims to characterize zero-RL behavior and inform future improvements.Source: §5 · p. 12
  • Zero-RL variants target training mechanisms and stability: DAPO adds four techniques [7], Dr. GRPO removes length and standard-deviation penalties [20], and VAPO builds on PPO [28].LUFFY [9] instead combines on-policy training with off-policy reasoning demonstrations to balance imitation and exploration.Source: §5 · p. 12
  • GHPO addresses reward sparsity by using all training data through adaptive difficulty detection and prompt refinement, unlike DAPO’s filtering of very easy or hard prompts [7].The passage characterizes filtering as potentially discarding valuable training data.Source: §5 · p. 13
  • Curriculum learning gradually introduces harder samples, while DAPO [7] filters prompts with perfect or zero accuracy; filtering can discard valuable data.The passage frames reward sparsity as a persistent challenge, especially in complex reasoning where correct solutions are rare.Source: §5 · p. 13

6 Conclusion

The conclusion frames instability and inefficiency in RLVR as problems associated with capacity-difficulty mismatch and sparse rewards. GHPO adaptively refines prompts to guide harder problems and support exploration on more manageable ones, and the paper reports gains across six mathematics benchmarks.

6 Conclusion

  • Approximately 5% average performance gain across six challenging mathematics benchmarks is reported for GHPO.The conclusion also reports enhanced training stability.Source: §6 · p. 13
  • GHPO uses adaptive prompt refinement to balance imitation learning on challenging problems with exploration-based RL on more manageable tasks.The framework dynamically calibrates task difficulty.Source: §6 · p. 13
  • The conclusion identifies capacity-difficulty mismatch and sparse rewards as central challenges behind RLVR training instability and inefficiency.It presents GHPO as a scalable, data-efficient approach for adapting learning to model capabilities.Source: §6 · p. 13

A Supplementary Case

The supplementary case works through a double sum by telescoping its inner and outer sums, then reducing the outer sum with the substitution k = i + j.

A Supplementary Case

  • The double sum’s inner expression is simplified by decomposition into terms that telescope.The derivation says most terms cancel in the inner sum.Source: A Supplementary Case · p. 17
  • Setting k = i + j reduces the double sum to a single sum, with k + 1 pairs of non-negative integers contributing for each fixed k.The count of pairs is used to rewrite the sum’s numerator factor.Source: A Supplementary Case · p. 18
  • The outer series is evaluated by telescoping, yielding 1/210 for the single series and a final double-sum value of 1/3600.The finite partial sum is given as Sn = 1/(5·6·7) − 1/((n+3)(n+4)(n+5)(n+6)(n+7)).Source: A Supplementary Case · pp. 17–18

Related papers

Loading 2507.10628v2…