PaperlayerView exact revision on arXiv

Source-linked AI summary

ExGRPO: Learning to Reason from Experience

Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F. Wong, Yu Cheng

arXiv:2510.02245v2cs.LGcs.AIcs.CL

TL;DR

On-policy RLVR discards rollout experience after one update, leaving the value and management of past reasoning experience underexplored. The paper identifies correctness and entropy as experience indicators, then introduces ExGRPO to organize replay and mix fresh exploration with past experience. Across five backbones, ExGRPO improves mathematical and general reasoning performance over on-policy RLVR and stabilizes training where on-policy methods fail.

  • Problem

    On-policy RLVR discards rollout experiences after a single update, while the value and management of past reasoning experience remain underexplored.

  • Method

    ExGRPO organizes replayed trajectories by rollout correctness, prioritizes low-entropy experiences, and combines them with fresh on-policy samples through a mixed-policy objective.

  • Results

    +3.5 and +7.6 points are the average ExGRPO gains over on-policy RLVR on in-distribution and out-of-distribution benchmarks, respectively.

  • Takeaways & Limitations

    Principled experience management improves reasoning performance and stabilizes RLVR training across weaker and stronger models where on-policy methods fail.

  • Takeaways & Limitations

    The evaluation focuses on verifiable mathematical and general reasoning tasks, so applicability to open-ended tasks with subjective and dense rewards remains an open question.

Abstract

from arXiv · show

Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm for improving the reasoning ability of large language models. However, standard on-policy training discards rollout experiences after a single update, leading to computational inefficiency and instability. While prior work on RL has highlighted the benefits of reusing past experience, the role of experience characteristics in shaping learning dynamics of large reasoning models remains underexplored. In this paper, we are the first to investigate what makes a reasoning experience valuable and identify rollout correctness and entropy as effective indicators of experience value. Based on these insights, we propose ExGRPO (Experiential Group Relative Policy Optimization), a framework that organizes and prioritizes valuable experiences, and employs a mixed-policy objective to balance exploration with experience exploitation. Experiments on five backbone models (1.5B-8B parameters) show that ExGRPO consistently improves reasoning performance on mathematical/general benchmarks, with an average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO stabilizes training on both stronger and weaker models where on-policy methods fail. These results highlight principled experience management as a key ingredient for efficient and scalable RLVR.

1 INTRODUCTION

RLVR on-policy training discards rollout experience after one update, motivating experience reuse for more efficient and stable reasoning optimization. The paper identifies valuable experience properties and introduces ExGRPO to manage and replay them.

  • On-policy RLVR discards valuable rollout experience after a single gradient update, wasting computation and learning opportunities.The paper frames experience reuse as a response to this on-policy inefficiency.
  • ExGRPO identifies rollout correctness and trajectory entropy as indicators of reasoning-experience value.Intermediate-difficulty tasks and low-entropy trajectories tend to benefit RLVR optimization.
  • ExGRPO organizes partially correct trajectories by correctness and prioritizes selected low-entropy experiences in a replay buffer.Its mixed-policy objective combines fresh exploration with strategically selected past experience.
  • +3.5 and +7.6 points are the average ExGRPO gains over on-policy RLVR on in-distribution and out-of-distribution benchmarks.The evaluation spans five Qwen and Llama backbone models from 1.5B to 8B parameters.
  • ExGRPO stabilizes training on weaker and stronger models where on-policy optimization collapses.The reported cases are Llama-3.1 8B and the stronger LUFFY model.

2 RELATED WORK

Prior RLVR work mainly uses on-policy optimization, while emerging off-policy approaches reuse historical or external data. The paper focuses on the underexplored quality of experiences stored in replay buffers.

  • Most RLVR methods use on-policy optimization, whereas newer approaches incorporate demonstrations, historical data, or direct off-policy updates.The cited approaches include policy gradients, SFT, knowledge distillation, and direct off-policy rules.
  • Existing RLVR replay approaches overlook data quality within the experience buffer, which is the central focus of this work.The paper distinguishes its focus from methods that primarily address distribution mismatch or replay itself.
  • In RLVR, an experience is a state-action-reward trajectory generated during rollout of a reasoning chain.The paper uses experience, trajectory, and rollout interchangeably.

3 PRELIMINARIES

The preliminaries define RLVR and GRPO, then analyze which questions and trajectories provide useful learning signals. Medium-difficulty questions and low-entropy trajectories emerge as the central selection guidelines.

  • 3.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD: RLVR assigns verifiable rewards by comparing an extracted model answer with a predefined golden answer.The reward is 1 when the output contains the correct final answer and 0 otherwise.
  • 3.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD: GRPO estimates each trajectory’s advantage by normalizing its reward against the empirical reward statistics of a sampled group.The method avoids requiring an additional value model.
  • 3.2 PRELIMINARY STUDY ON EXPERIENCE DATA: Training on medium questions produces the largest performance gains among models restricted to easy, medium, or hard online-correctness buckets.Buckets are defined as Easy [75%, 100%), Medium (25%, 75%], and Hard (0, 25%].
  • 3.2 PRELIMINARY STUDY ON EXPERIENCE DATA: Correct reasoning trajectories exhibit lower entropy than incorrect trajectories, supporting entropy as an online quality proxy.An external Qwen3-32B judge is used to assess logical validity among correct outputs.
  • 3.2 PRELIMINARY STUDY ON EXPERIENCE DATA: Medium-difficulty questions provide the most valuable optimization signals, while entropy minimization is an effective trajectory-selection heuristic.These guidelines motivate the experience-management design.
  • 3.2 PRELIMINARY STUDY ON EXPERIENCE DATA: High-entropy replayed trajectories can produce misguided learning and systematic reasoning errors, including invalid code blocks in math solutions.The paper calls this contamination a snowball effect.

4 METHODOLOGY

ExGRPO manages replayed reasoning experiences through collection, partitioning, and selection, then combines them with fresh on-policy samples. Its optimization mechanisms address distribution shift while preserving exploration and stability.

  • 4.1 EXGRPO: EXPERIENCE MANAGEMENT: Successful rollouts are stored with each question’s correctness rate, then partitioned into difficulty buckets for replay.Questions solved in every rollout enter a Retired Set so optimization focuses on partially solved questions.
  • 4.1 EXGRPO: EXPERIENCE MANAGEMENT: Experience selection samples questions near 50% correctness and chooses the lowest-entropy trajectory for each selected question.This operationalizes the paper’s medium-difficulty and entropy-based guidelines without an external judge.
  • 4.2 EXGRPO: EXPERIENTIAL POLICY OPTIMIZATION: ExGRPO unifies on-policy exploration with off-policy replay in a joint objective, using a tunable experiential proportion ρ.Mini-batches contain fresh samples and replayed samples, with a lower bound on on-policy data when the buffer is depleted.
  • 4.2 EXGRPO: EXPERIENTIAL POLICY OPTIMIZATION: Replay optimization forms a mixed advantage group containing one past-policy trajectory and new current-policy rollouts, then reweights the replayed trajectory.The reweighting corrects the distribution shift between past and current policies.
  • 4.2 EXGRPO: EXPERIENTIAL POLICY OPTIMIZATION: Policy shaping transforms replay importance weights with f(x) = x/(x+β), using β = 0.1 to amplify low-probability signals and dampen high-probability ones.The transformation encourages learning from novel aspects of replayed trajectories.
  • 4.2 EXGRPO: EXPERIENTIAL POLICY OPTIMIZATION: ExGRPO delays replay until a training batch exceeds a predefined Pass@1 threshold, avoiding low-quality early experiences.The on-policy process runs first while the model’s capabilities are still developing.

5 EXPERIMENTS

Experiments evaluate ExGRPO across mathematical and out-of-distribution benchmarks, model variants, and a continuous-reward extension, with comparisons against on-policy RLVR.

  • 5.1 EXPERIMENTAL SETUP: Evaluation covers six in-distribution mathematics benchmarks and three out-of-distribution reasoning tasks, using Avg@32 for AIME and AMC and Pass@1 elsewhere.The setup spans AIME24/25, AMC, MATH-500, Minerva, OlympiadBench, ARC-c, GPQA-Diamond, and MMLU-Pro.
  • 5.1 EXPERIMENTAL SETUP: The experiments use five backbone configurations, including Qwen and Llama models from 1.5B to 8B parameters, with replay data comprising 50% of each mini-batch.ExGRPO is generally activated after batch Pass@1 exceeds 35%, except for Llama because on-policy RLVR collapses.
  • 5.2 MAIN RESULTS: +3.0 points over on-policy RLVR on Qwen2.5-Math 7B math reasoning tasks, with the advantage also holding on out-of-distribution benchmarks.The comparison is reported in Table 1 and the accompanying results discussion.
  • 5.2 MAIN RESULTS: ExGRPO improves performance across backbone scales and initializations, including up to +5.3 points on Qwen2.5-Math 1.5B Base and +4.1 points on Qwen2.5-7B Instruct.The method also enables successful Llama-3.1 base training where on-policy RLVR collapses.
  • 5.3 EXTENSION TO RL WITH CONTINUOUS REWARDS: The continuous-reward extension replaces rollout success rates with normalized reward variance and preferentially selects queries with moderate variance.This extension is evaluated preliminarily on WildChat-IF with Qwen2.5-7B-Base and a preference model.

6 ANALYSIS AND DISCUSSION

The analysis shows that experience replay stabilizes training, improves data efficiency, and works best when selection and replay balance exploitation with exploration.

  • 6.1 TRAINING DYNAMICS: Replay stabilizes weaker-model training by reusing early correct solutions, whereas on-policy learning can collapse through weak rewards, lost exploration, and entropy explosion.For Llama3.1-8B, replay guides exploration toward the model’s current capability and prevents premature collapse.
  • 6.1 TRAINING DYNAMICS: About half of the dataset eventually achieves at least one successful trajectory (Pass@8), showing that revisiting past successes improves data utilization for stronger models.The retired set grows as capability improves, further increasing experience efficiency.
  • 6.1 TRAINING DYNAMICS: Replay efficiency depends on sample value rather than buffer size: repeated easy questions waste samples, while larger replay ratios can over-exploit and degrade performance below on-policy training.The best reported balance is ρ = 50%; ρ = 75% stifles exploration, whereas ρ = 25% accumulates experiences without comparable gains.
  • 6.2 ABLATION STUDIES: ExGRPO’s analytically derived question and trajectory selection heuristics consistently outperform random selection, while policy shaping prevents exploitation from hindering exploration.Shaping alone does not improve on-policy RLVR, supporting the combined design.
  • 6.3 SENSITIVITY ANALYSIS: Sensitivity analysis evaluates narrower and broader Gaussian difficulty sampling, plus a hard-biased variant, to test how sampling focus affects performance.The accompanying table identifies the best setting for each reported column, but the supplied passages do not provide its cell values.

7 CONCLUSION

The conclusion presents principled experience management as a way to improve reasoning performance and stabilize RLVR training across models and benchmarks.

  • 7 CONCLUSION: ExGRPO addresses limited systematic understanding of experience value by managing, selecting, and replaying high-quality experiences.Experiments across multiple backbones and benchmarks show consistent improvement and stabilization where on-policy RLVR fails.

ETHICS STATEMENT

The paper states that its datasets, methods, and models were designed without sensitive personal data or content likely to introduce societal harm.

  • ETHICS STATEMENT: The work reports no sensitive personal data or content likely to introduce societal bias, harm, or discrimination.The authors describe the datasets, methodologies, and model as ethically sound.

A LIMITATIONS

The paper acknowledges scope, heuristic, algorithm-family, and future-extension limitations while documenting ExGRPO’s replay and mixed-policy formulation.

  • A LIMITATIONS: Evaluation is limited to verifiable mathematical and general reasoning tasks, leaving applicability to subjective open-ended tasks unresolved.Correctness-based bucketing may not directly transfer to creative writing or other dense-reward settings.
  • A LIMITATIONS: The heuristic definition of valuable experience may overlook useful incorrect trajectories and risk premature convergence in some scenarios.The limitation concerns ExGRPO’s emphasis on exploiting selected experiences.
  • A LIMITATIONS: ExGRPO’s interaction with other families of RL algorithms has not been explored.The paper identifies multimodal reasoning and agentic reinforcement learning as future extensions.
  • D.1 RECAP AND FORMULATION: ExGRPO partitions replay experiences by rollout correctness, samples buckets with Gaussian weighting, and selects the lowest-entropy trajectory for experiential training.Remaining batch positions use on-policy samples, creating a mixed experiential and on-policy objective.
  • D.2 UNBIASEDNESS OF EXPERIENTIAL GRADIENT: Exact importance weighting makes the replayed contribution unbiased relative to the corresponding on-policy term, even when advantages depend on mixed-group statistics.The result assumes exact per-token importance ratios and the stated assumption A1.

D.3 VARIANCE DECOMPOSITION AND BOUNDS

The variance analysis identifies importance-weighted experiential gradients as the key source of uncertainty and derives general and tighter bounds under progressively stronger assumptions. It motivates low-entropy selection, importance correction, and policy shaping as controls for this variance.

  • D.3 VARIANCE DECOMPOSITION AND BOUNDS: ExGRPO’s experiential variance is bounded under finite second moments and bounded trajectory importance ratios, without requiring independence among group members.A tighter bound additionally assumes pairwise uncorrelated contributions and zero covariance with the replayed term.
  • D.3 VARIANCE DECOMPOSITION AND BOUNDS: Removing group standard-deviation normalization reduces denominator-induced coupling and typically lowers the per-trajectory second moment in the variance bounds.The implementation mean-centers advantages without the group standard-deviation term.
  • D.3 VARIANCE DECOMPOSITION AND BOUNDS: Policy shaping maps importance weights through a bounded monotone transform, reducing extreme-weight variance while introducing bias.For small weights it can amplify signals, whereas large weights are damped toward a maximum of 1.
  • D.3 VARIANCE DECOMPOSITION AND BOUNDS: The variance is governed by the second moment E[W 2U 2], making importance-weight control central to stable replay.Exact trajectory-level importance weighting preserves unbiasedness, while controlling the importance term tightens the variance bound.

E.4 ABLATION AND ADDITIONAL RESULTS

Ablations show that ExGRPO’s selection, replay-ratio, and policy-shaping choices each contribute to performance, while matched comparisons favor ExGRPO over RePO. Replaying fully solved queries is less useful than retaining informative mid-difficulty experiences.

  • Experience Ratio: 48.3 average performance with the default ρ=50% exceeds 46.4 at ρ=25% and 44.8 at ρ=75%.The intermediate replay ratio balances past experience with fresh on-policy learning.
  • Policy Shaping Dynamics: 41.2 average performance without policy shaping shows that shaping is essential for preserving exploration during experience exploitation.Without shaping, entropy initially falls below the on-policy baseline and the model fails to sustain exploration.
  • vs. RePO: 52.3 versus 46.8 average out-of-distribution performance: ExGRPO outperforms RePO under identical training conditions.The comparison matches RePO’s prompt template, data sources, configuration, and on-policy rollout count.
  • Effects of Reintroducing Retired Queries: Reintroducing 10% of retired, fully solved queries every 25 steps lowers both in-distribution and out-of-distribution performance.The replayed solved queries provide limited learning signal while displacing more informative mid-difficulty cases.

F.3 ALTERNATIVE SELECTION METRICS

The paper compares perplexity and entropy as online indicators of reasoning quality. Both are lower for correct trajectories across correctness buckets, but ExGRPO selects entropy as its operational metric.

  • F.3 ALTERNATIVE SELECTION METRICS: Correct reasoning trajectories consistently have lower perplexity and entropy than incorrect trajectories across all correctness buckets.Figure 10 compares these internal metrics against external CoT-judge correctness labels.
  • F.3 ALTERNATIVE SELECTION METRICS: ExGRPO adopts entropy rather than perplexity as its trajectory-quality metric.The paper states that the choice is justified by the analysis of these internal signals against external correctness judgments.

F.4 SNOWBALL EFFECTS

The snowball-effect analysis links high-entropy, code-containing reasoning trajectories with weaker reasoning quality. Replay can therefore reinforce flawed reasoning unless experience selection favors lower-entropy, more coherent trajectories.

  • F.4 SNOWBALL EFFECTS: High-entropy trajectories can create a snowball effect by repeatedly replaying flawed reasoning and entrenching errors in the learned policy.The paper highlights unnecessary or invalid code blocks as a recurring source of deficient reasoning chains.
  • F.4 SNOWBALL EFFECTS: 0.14 versus 0.07 entropy for medium-difficulty problems: code-containing trajectories are more uncertain than trajectories without code.Across difficulty levels, code use is also associated with a lower proportion of logically correct CoT sequences.
  • F.4 SNOWBALL EFFECTS: Among trajectories reaching the same correct answer, the lowest-entropy example is shorter and coherent, whereas the highest-entropy example uses longer, logically unsound code verification.The external CoT judge also identifies the high-entropy trajectory as poor reasoning.
Loading 2510.02245v2…