PaperlayerView exact revision on arXiv

Source-linked AI summary

RePO: Replay-Enhanced Policy Optimization

Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, Chaochao Lu

arXiv:2506.09340v1cs.CLcs.AIcs.LG

TL;DR

GRPO relies on multiple on-policy outputs per prompt, which raises computational costs and can yield no gradient signal when rewards are uniform. RePO adds replayed off-policy samples to broaden the outputs used for optimization. Across experiments, it outperforms GRPO on mathematical reasoning, with reported gains for three models.

  • Problem

    GRPO requires multiple on-policy samples per prompt, and when all samples receive the same reward its estimated advantages collapse to zero, providing no meaningful gradient signal.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    RePO combines on-policy updates with off-policy outputs retrieved from a replay buffer using diverse replay strategies.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    RePO outperforms GRPO on mathematical reasoning, with absolute average accuracy gains of 18.4, 2.0, and 4.1 points for Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-1.7B, respectively.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors report RePO’s effectiveness across five LLMs, seven mathematical reasoning benchmarks, and six general reasoning tasks.

    Source: §6 · p. 8

    Experiments cover LLMs up to 7B parameters; larger models remain unexplored due to computational constraints.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) is vital for optimizing large language models (LLMs). Recent Group Relative Policy Optimization (GRPO) estimates advantages using multiple on-policy outputs per prompt, leading to high computational costs and low data efficiency. To address this, we introduce Replay-Enhanced Policy Optimization (RePO), which leverages diverse replay strategies to retrieve off-policy samples from a replay buffer, allowing policy optimization based on a broader and more diverse set of samples for each prompt. Experiments on five LLMs across seven mathematical reasoning benchmarks demonstrate that RePO achieves absolute average performance gains of $18.4$ and $4.1$ points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively, compared to GRPO. Further analysis indicates that RePO increases computational cost by $15\%$ while raising the number of effective optimization steps by $48\%$ for Qwen3-1.7B, with both on-policy and off-policy sample numbers set to $8$. The repository can be accessed at https://github.com/SihengLi99/RePO.

1 Introduction

GRPO’s reliance on multiple on-policy outputs creates computational overhead and can leave optimization without a gradient signal when rewards are uniform. RePO combines on-policy updates with replayed off-policy samples, and experiments report higher mathematical-reasoning performance than GRPO.

1 Introduction

  • GRPO’s multiple on-policy samples per prompt add computational overhead, while uniform rewards can collapse estimated advantages to zero and remove the gradient signal.Source: §1 · p. 1
  • RePO combines on-policy and off-policy updates using replay strategies to retrieve prior outputs, broadening per-prompt samples and improving data efficiency while reducing overfitting.Strategies can prioritize high-reward outputs or outputs closer to the current policy.Source: §1 · p. 1
  • RePO outperforms GRPO, with absolute average accuracy gains of 18.4, 2.0, and 4.1 points for Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-1.7B, respectively.The experiments cover five open-source LLMs; the passage also reports stronger generalization on general reasoning benchmarks.Source: §1 · p. 1
  • Figure 1 depicts policy updates from both current on-policy samples and replay-buffer samples, with advantage estimation performed separately for each.Source: Figure 1 · p. 2

2 Preliminary

The preliminary section introduces PPO and its advantage estimation, then describes GRPO’s group-based reward normalization and its limitations.

2.1 Reinforcement Learning

  • PPO uses an importance-sampling ratio to constrain policy updates, while token-level advantages guide updates; GAE relies on a large value model and adds memory and computational overhead.Source: §2.1 · p. 2

2.2 Group Relative Policy Optimization

  • GRPO estimates each output’s advantage from rewards across multiple outputs sampled for the same prompt, but requires multiple on-policy samples and can lose its optimization signal when rewards are identical.This zero-advantage case is especially relevant on overly simple or difficult tasks, or when policy outputs lack diversity.Source: §2.2 · p. 2

3 Replay-Enhanced Policy Optimization

RePO extends GRPO with on-policy and replay-buffer off-policy updates that reuse previously sampled outputs. Its retrieval strategies offer flexibility in selecting past samples for optimization.

3 Replay-Enhanced Policy Optimization

  • RePO combines an on-policy update with an off-policy replay-buffer update that reuses previously sampled outputs.The replay strategy determines which prior samples are retrieved.Source: §3 · p. 2, §3.2 · p. 3

3.1 On-Policy Update

  • After each on-policy update, RePO stores sampled outputs and their generation probabilities in a replay buffer for later off-policy updates.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Off-Policy Update

  • Clipping constrains divergence from the behavior policy, while importance weighting reduces the influence of replay samples unlikely under the current policy.Source: §3.2 · p. 3
  • RePO retrieves prior outputs and their generation probabilities using full-scope, recency-based, reward-oriented, or variance-driven strategies.Full-scope retrieves all prior samples; recency-based takes the most recent K; reward-oriented selects highest-reward samples; variance-driven selects samples with highest reward variance.Source: §3.2 · p. 3

3.3 Advantage Estimation

  • RePO estimates advantages separately for on-policy and off-policy experiences, a strategy described as reducing potential interference between their updates.Source: §3.3 · p. 4

4 Experiments

The experiments evaluate RePO against GRPO on mathematical and general reasoning benchmarks and analyze replay strategies, sample counts, and optimization efficiency. RePO performs better on average, while analyses show that results vary with replay design and sample setting.

4.3 Main Results

4.1 Evaluation Setup; 4.2 Implementation

  • Experiments use five Qwen models, 1,024 DeepMath training problems, seven math benchmarks, and five general-reasoning benchmarks.The evaluation also reports avg@32 for AIME24, AIME25, and AMC, and pass@1 for the other math benchmarks.Source: §4.2 · pp. 4–5, §4.1 · p. 4

4.3 Main Results

  • Replay strategy is model- and task-dependent; base models use Recency, instruct models Reward-oriented, while prior outputs are proposed to diversify training and mitigate overfitting.The authors maintain this strategy assignment across RePO experiments.Source: §4.3 · p. 5

4.4 Analysis

  • On Qwen3-1.7B, Split advantage estimation beats Mixed by 7.7 and 16.4 average points under different sample settings.Split estimates advantages separately for on-policy and off-policy samples; the authors use it throughout the work.Source: Table 4 · p. 6, §4.4 · pp. 6–7
  • When applied to Dr. GRPO, RePO outperforms the original Dr. GRPO, achieving absolute average performance gains of 1.3 and 4.2 points for Qwen2.5-Math-1.5B and Qwen3-1.7B, respectively.Source: §4.4 · p. 6
  • Recency-based and Reward-oriented replay consistently outperform alternatives; accuracy first rises, then falls as off-policy sample counts increase.The authors attribute the decline at high counts to noise from the finite pool of off-policy samples.Source: §4.4 · p. 7, Figure 2 · p. 7
  • RePO costs 15% more computation yet gains 4.8 and 4.1 average-performance points over GRPO across seven math benchmarks.The comparison is on Qwen3-1.7B across different sample settings; training time is normalized to GRPO with four on-policy samples.Source: §4.4 · pp. 7–8, Table 6 · p. 8
  • RePO raises effective steps by 10.0% and 47.8% over GRPO when on-policy and off-policy sample counts are both 4 and 8, respectively.The analysis assumes each step contains one prompt and optimizes only over its sampled outputs.Source: Table 7 · p. 8, §4.4 · p. 8

5 Related Work

Earlier RL methods address LLM optimization with different advantage-estimation strategies, while GRPO uses multiple outputs per prompt. RePO extends GRPO with replayed outputs from previous iterations rather than samples from more advanced models.

5 Related Work

  • GRPO estimates advantages from multiple outputs per prompt, while ReMax, RLOO, and REINFORCE++ use different reward baselines.ReMax uses a single greedy output, RLOO uses the mean reward of other outputs, and REINFORCE++ uses the global-batch mean reward.Source: §5 · p. 8
  • RePO retrieves off-policy samples generated in previous iterations, avoiding reliance on more advanced models while increasing policy-optimization flexibility.It also introduces replay strategies for selecting suitable samples.Source: §5 · p. 8
  • Prior work identifies GRPO length and difficulty biases and entropy collapse, while LUFFY incorporates off-policy samples from more advanced models.RePO instead uses samples from its own previous iterations.Source: §5 · p. 8

6 Conclusion

RePO retrieves previously sampled off-policy outputs to optimize policies with a broader set of outputs per prompt. Experiments across five LLMs, seven mathematical reasoning benchmarks, and six general reasoning tasks validate its effectiveness.

6 Conclusion

  • RePO retrieves previously sampled off-policy outputs for current policy optimization, using a broader set of outputs per prompt.The paper describes this approach as reducing overfitting and increasing flexibility.Source: §6 · p. 8
  • Experiments across five LLMs, seven mathematical reasoning benchmarks, and six general reasoning tasks validate RePO's effectiveness.Source: §6 · p. 8
  • The authors expect RePO to contribute to reinforcement-learning optimization for advancing LLMs.Source: §6 · p. 8

Limitations

The reported evaluation is limited to LLMs with up to 7B parameters, and several training hyperparameters were not thoroughly investigated because of computational cost.

Limitations

  • Models larger than 7B parameters remain unexplored because of computational constraints.Assessing RePO on more capable models is identified as a valuable direction.Source: Limitations · p. 9
  • The authors identify further investigation of model scale and training hyperparameters as needed.Source: Limitations · p. 9
  • Learning rate, replay-sample count, clipping ratio, and the off-policy-update coefficient were not thoroughly investigated because of computational cost.The paper calls for further analysis of these factors.Source: Limitations · p. 9

A.1 Training Details

RePO applies off-policy updates only in the final of three epochs, with eight on-policy and eight off-policy samples per step. The experiments use a global batch size of 32 and eight NVIDIA A100 GPUs.

A.1 Training Details

  • Both GRPO and RePO train for three epochs under identical configurations, but RePO applies off-policy updates only in the final epoch.Source: §A.1 · p. 11
  • RePO uses eight on-policy and eight off-policy samples per step, increasing examples per step from 32 to 64 in its final epoch.The global batch size is 32.Source: §A.1 · p. 11
  • Training uses a 1e-6 learning rate with cosine decay, prompt and completion limits of 512 and 1024 tokens, and eight A100 GPUs.Four GPUs are allocated to policy optimization and four to VLLM output sampling; Math-Verify2 assigns rewards of 1 for correct final answers and 0 otherwise.Source: §A.1 · p. 11
Loading 2506.09340v1…