PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient RL Training for LLMs with Experience Replay

Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, Remi Munos

arXiv:2604.08706v1cs.LG

TL;DR

LLM post-training largely favors fresh on-policy data, although rollout generation is expensive and replay buffers remain underexplored. The paper combines theoretical analysis with empirical evaluation to design replay buffers around staleness, diversity, and generation cost. It finds that suitable replay can reduce compute by up to 40% while maintaining or surpassing on-policy accuracy, with validation limited to the evaluated model scales.

  • Problem

    LLM post-training largely discards rollouts after one update because off-policy degradation is thought to outweigh replay's computational benefits, despite rollout generation often dominating the training budget.

  • Method

    The paper combines theoretical bias-variance analysis with empirical studies of replay-buffer size, replay ratio, sampling, and training stability in asynchronous LLM RL.

  • Results

    Up to 40% of the compute budget can be saved while maintaining, and sometimes surpassing, the final accuracy of on-policy baselines.

  • Takeaways & Limitations

    When rollout generation is sufficiently costly, compute-efficient LLM RL can favor replay over strictly on-policy sampling without degrading final accuracy.

  • Takeaways & Limitations

    The reported efficiency gains were evaluated at the model scales studied, and further validation on larger frontier models is needed.

Abstract

from arXiv · show

While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.

1 Introduction

LLM RL training often discards generated rollouts after one update, despite inference dominating post-training costs. This work studies replay buffers as a controlled trade-off among staleness, diversity, and compute, finding that suitable buffers can improve compute efficiency while preserving performance and stability.

  • Motivation: Inference can consume more than 80% of post-training GPU hours, while PPO and GRPO typically discard rollouts after a single gradient update.This creates substantial sample inefficiency in LLM RL pipelines.
  • Motivation: Experience Replay stores and reuses trajectories, challenging the prevailing assumption that off-policy degradation outweighs its computational benefits in LLM training.The paper contrasts LLM practice with replay's established role in classical RL.
  • Approach: Replay buffers trade controlled staleness and reduced diversity for lower inference costs, with optimal efficiency balancing data freshness, diversity, and generation expense.The paper formalizes this balance through a bias-variance analysis of stochastic gradient descent.
  • Empirical findings: Well-sized buffers stabilize training and preserve output diversity, whereas aggressive sample reuse can degrade performance and pass@k metrics.The buffer can act as a regularizer when its configuration is appropriate.
  • Empirical findings: Up to 40% of the compute budget can be saved while maintaining, and sometimes surpassing, the on-policy baseline's final accuracy.The paper reports these gains for simple buffer strategies and illustrates them in Figure 1.
  • Implication: The study reframes efficient RL fine-tuning around maximizing performance per unit of compute rather than performance per training step.It also explores sophisticated sampling strategies and alternative losses for further efficiency gains.

2 Related Work

Prior work established replay buffers as tools for stability and sample efficiency in deep RL, while LLM reasoning pipelines largely remained on-policy. Recent LLM replay studies pursue exploration, data quality, or accuracy, whereas this work emphasizes compute efficiency.

  • Experience Replay in RL: Replay buffers underpin stability and sample efficiency in deep RL algorithms including DQN, Soft Actor-Critic, and DDPG.Prior techniques also include prioritized and hindsight experience replay.
  • Experience Replay in RL: Modern LLM reasoning pipelines commonly discard trajectories after one gradient update to avoid off-policy degradation, despite some unavoidable off-policiness in practice.The related-work discussion contrasts this convention with classical deep RL.
  • Replay Buffers for LLMs: Recent LLM replay studies use buffers to enhance exploration, final performance, or data quality through specialized losses, filtering, dynamic sampling, or multi-phase training.These approaches differ in motivation and design from the present compute-efficiency focus.
  • Replay Buffers for LLMs: This work focuses strictly on compute efficiency rather than proposing a new training paradigm to surpass state-of-the-art accuracy.The paper positions its contribution as a study of replay-buffer efficiency in LLM training.

3 Experience Replay for Off-Policy RL

Replay buffers reuse generated trajectories in asynchronous LLM RL, reducing inference compute while trading off staleness and sample diversity. Buffer size and the inference-worker-to-trainer ratio jointly control these effects and the resulting efficiency–accuracy trade-off.

  • Replay-buffer implementation: Replay buffers store trajectories without removing sampled items, allowing rollout-generation costs to be amortized across multiple training uses.Inference workers add rollouts to a buffer while trainers continuously sample from it.
  • Experimental design: For the illustrated configurations, average replay ratios are 1.78, 3.42, and 7.0 for (W, T) equal to (6, 2), (5, 3), and (4, 4), respectively.Figure 2 compares distributions of off-policiness, replay ratio, and steps-since-last-use across buffer sizes and worker–trainer ratios.
  • Design dimensions: Buffer design and the W/T ratio jointly determine compute efficiency, off-policiness, and sample diversity.The ratio compares inference workers with trainers, while buffer size and sampling behavior shape reuse and staleness.
  • Compute–accuracy trade-off: Lowering W/T makes gradient steps cheaper because more trainers reuse each generated rollout, but also makes training more off-policy and less diverse.The resulting design problem is to maximize accuracy achievable at a given compute cost.
  • Off-policiness and diversity: Larger buffers increase average off-policiness but can improve local diversity by reducing repeated use of the same samples in close succession.At fixed W/T, global reuse can remain similar while samples are less likely to be reused consecutively.
  • Practical design: The study prioritizes simple buffer implementations that require only modest departures from current state-of-the-art pipelines.The motivating objective is increased efficiency with preserved accuracy.

4 Mathematical Analysis

The analysis models replay-buffer training as non-convex stochastic optimization with replay-induced bias, variance, and sample correlation. It derives an optimal design showing when replay is favored by expensive rollout generation and when on-policy training remains preferable.

  • Analytical framework: The theory studies convergence toward stationary points using the decay of the expected squared gradient norm in a classical non-convex stochastic optimization framework.The objective is to characterize how replay-buffer design affects learning efficiency.
  • Bias and dependence: Experience replay introduces gradient-estimate bias through correlations between replayed samples and the current parameter iterates, even with importance-ratio correction.The bias is expected to increase when buffered trajectories strongly influenced subsequent parameter updates.
  • Variance: Gradient-estimate variance grows with off-policiness and can also reflect statistical dependence among samples drawn within the same minibatch.The analysis models both per-sample variance and within-minibatch correlation.
  • Limitation: Importance sampling corrects marginal distribution mismatch but does not generally make the parameter-conditioned previous policy distribution computable.This limits straightforward correction of replay-induced dependencies.
  • Optimal design: The optimal replay-buffer design is expressed through the staleness horizon N/R and replay ratio B/R under an asymptotically large compute budget.The theorem optimizes the convergence bound over learning rate, buffer size, replay count, and batch size.
  • Design trade-offs: Replay becomes optimal when rollout generation is expensive or off-policy effects are small, whereas the optimal staleness horizon approaches zero when rollout cost is small or variance and correlation are high.The result formalizes a three-way trade-off among staleness-induced noise, replayed-sample correlation, and rollout-versus-training compute imbalance.

5 Experimental results

Experiments show that replay buffers can improve compute efficiency while preserving or improving accuracy and output diversity, with buffer size and replay ratio governing stability and performance. Refinements such as positive-bias sampling and alternative losses can further improve the basic buffer design, though larger-scale validation remains necessary.

  • Experimental setup: Replay buffers are evaluated in asynchronous RL fine-tuning of Qwen models with GRPO on OpenR1-Math-220k and evaluation on OpenR1-Math-220k or MATH.The setup uses W inference workers, T trainers, and usually uniform sampling from a shared buffer.
  • Main results: Up to 40% of compute can be saved to reach a given accuracy, while replay can outperform strictly on-policy training across compute budgets.Replay also stabilizes training, sometimes preventing crashes and enabling higher peak accuracy.
  • Buffer size and off-policiness: Larger buffers slow training but stabilize it and can yield higher maximal accuracy reached later in the run.Increasing buffer size also increases average sample off-policiness.
  • Replay ratio: Replay ratios of 2.2, 5.6, and 17.6 arise for worker-trainer ratios (6, 2), (5, 3), and (4, 4), with aggressive replay eventually degrading performance.Moderate replay ratios do not adversely affect maximal accuracy, while more extreme configurations can remain attractive because of compute efficiency.
  • Output diversity: Experience replay improves pass@k for k > 1 and can preserve output diversity, with larger-k pass@k benefiting more than pass@1.The paper attributes this to increased diversity in the training distribution from older samples.
  • Main results: Reducing W/T improves compute efficiency but worsens learning dynamics, whereas larger buffers slow training while stabilizing it and preserving output diversity.Suitable configurations combine these effects into net improvements across metrics relative to strictly on-policy RL.
  • Further optimization: Positive-bias sampling retains fresh rollouts and additional fresh correct rollouts, while AsymRE replaces GRPO as an alternative loss.The refinements target greater stability and efficiency beyond the simplest uniform buffer.
  • Further optimization: Both positive-bias sampling and the alternative-loss variant substantially improve over the basic buffer implementation, but larger-scale experiments are needed.Uniform sampling without replacement produced inconclusive results in a separate ablation.

6 Conclusion

The conclusion presents replay buffers as a compute-efficiency mechanism for LLM reinforcement learning, balancing staleness, diversity, and rollout-generation cost. It reports up to 40% compute savings without sacrificing, and sometimes surpassing, on-policy accuracy, while identifying frontier-model validation as an open limitation.

  • Conclusion: Replay buffers create a three-way trade-off among staleness, sample diversity, and the relative cost of inference.The theoretical framework predicts that higher rollout-generation costs shift the optimal strategy toward experience replay.
  • Conclusion: A simple replay buffer can reduce compute by up to 40% while maintaining or surpassing the accuracy of on-policy baselines.The conclusion frames performance per unit of compute as a more practical objective than performance per gradient step.
  • Conclusion: Experience replay is presented as a key component for achieving compute-efficient RL pipelines.This conclusion follows the reported compute savings and accuracy outcomes.
  • Limitations: The efficiency gains have been validated only on the model scales evaluated, so their robustness on larger frontier models remains unconfirmed.The authors identify validation on larger frontier models as needed future work.

A.3 Experience Replay in Modern LLM Training

Modern LLM training has begun exploring replay mechanisms, but this work distinguishes itself by studying simple replay buffers in standard asynchronous pipelines for compute-efficiency trade-offs. Its central question is how much compute data reuse can save without degrading performance.

  • Prior work: Although on-policy methods dominate current LLM training, recent work has begun exploring replay mechanisms with different objectives.These objectives include exploration, successful-solution retention, and complex multi-phase or dynamic-sampling pipelines.
  • This work: Unlike prior approaches, this work analyzes a simple replay buffer added to standard pipelines rather than introducing complex objectives to maximize final accuracy.The focus is the efficiency trade-off in a standard asynchronous setup.
  • This work: The study asks how much compute can be saved by reusing data in a standard asynchronous setup without degrading performance.This frames replay as a compute-efficiency intervention rather than primarily an exploration mechanism.

B Pseudo-Code Implementation

The asynchronous implementation uses concurrent inference workers and trainers, contrasting a standard stream-based pipeline with a replay-buffer architecture.

  • Implementation: The pseudo-code simulates concurrent inference workers and trainers using Python’s asyncio library.Inference workers generate rollouts while trainers perform optimization steps.
  • Implementation: The implementation highlights the transition from stream-based processing to a shared replay-buffer architecture.This connects the pseudo-code to the paper’s central design change.

B.1 Queue-based Data Transfer

The asynchronous pipeline uses queues to transfer rollouts from inference workers to trainers, discarding samples after consumption. A replay buffer changes this transfer structure so trajectories remain available for repeated random sampling.

  • Queue-based transfer: Baseline asynchronous RL uses a LIFO pipe that prioritizes fresh samples and discards each trajectory after one update.This couples rollout generation tightly to training consumption.
  • Queue-based transfer: Inference workers continuously generate trajectories, push them into shared storage, and may generate under changing weights during concurrent updates.Weights can be updated concurrently with rollout generation, so later tokens may use different parameters than earlier tokens.
  • Queue-based transfer: Trainer workers pull batches from storage, compute GRPO/PPO losses and gradients, and update weights concurrently with sampling.The trainer loop operates independently from the sampler loop.
  • Replay-buffer modification: A replay buffer stores up to N trajectories, retains sampled items, and enables multiple samples of the same trajectory through random sampling.Implementing replay requires replacing the transfer queue with a buffer data structure.
  • Analytical modeling: The mathematical analysis separates replay effects into bias, variance, and dependence terms associated with stale and repeatedly reused samples.Reuse increases distributional dissimilarity between the generating and current policies, while buffer elements can become statistically coupled through prior updates.

C.2 Proof of Convergence

The convergence analysis reduces replay-buffer design to balancing compute cost against staleness, replay, variance, and dependence through normalized design ratios. It then derives optimality conditions and closed-form solutions under a power-law variance profile.

  • Convergence decomposition: The analysis models convergence by separately bounding bias and variance contributions from replayed samples.The main inequality identifies the final two quantities as the bias and variance parts.
  • Normalized design: The design objective depends on the staleness horizon x = N/R and replay ratio y = B/R rather than independently on buffer size, batch size, and rollout rate.The staleness horizon is the maximum trajectory staleness, while the replay ratio is the average number of sample reuses.
  • Normalized design: The optimal design minimizes a two-variable objective J(x, y) over positive ratios after reducing the original three-variable parameterization.The reduction follows from homogeneity under jointly scaling B, N, and R.
  • Scope of solution: The asymptotic ratio characterization does not prescribe absolute batch, buffer, or rollout values under finite compute.Finite-time effects, integer constraints, and divisibility assumptions can break scale invariance and affect practical choices.
  • Power-law variance: Under a power-law variance profile, the optimal staleness horizon is obtained by solving a quadratic equation and selecting its positive root.The derivation explicitly simplifies the discriminant before giving the closed-form solution.
  • Optimality conditions: As rollout cost µ increases, the optimal strategy shifts toward larger staleness horizons and replay ratios.Figure 6 illustrates the corresponding optimal x∗ = N/R and y∗ = B/R.

D.3 Metrics

The paper evaluates replay and training behavior with compute and sample-use metrics under idealized parallel-compute assumptions. It defines steps-since-last-use to characterize temporal reuse of rollouts.

  • Compute assumptions: The compute abstraction assumes FLOPS correspond closely to GPU work time and ignores several implementation effects such as tensor parallelism and sharding.It also assumes continuous task parallelization and compute independent of GPU count.
  • Sample-use metrics: Each rollout may produce zero, one, or several gradient-batch samples depending on how often the replay buffer samples it.The paper distinguishes a rollout as a sequence from each sampled occurrence used in gradient descent.
  • Sample-use metrics: Steps-since-last-use orders all training samples and records whether each occurrence is new or how many gradient steps elapsed since its preceding occurrence.The resulting values are summarized as histograms over samples from each trajectory.
  • Evaluation metric: Pass@k is summarized by taking the maximum of the median training curve across seeds and reporting the IQR at that maximum.The maximizing training step can differ across choices of k.

D.4 Buffer-specific aspects

Buffer-specific analysis estimates the relative rollout-generation cost, describes sharded buffers, and introduces positive-bias sampling as an alternative retention strategy. These choices determine how trajectories are stored and selected.

  • Compute-cost estimation: The compute ratio µ compares per-GPU training throughput with per-GPU inference throughput when the two processes operate independently without downtime.It is estimated from processed and generated rollout counts and refined using the median over random seeds.
  • Compute-cost estimation: The reported µ estimates are medians over independent runs, with interquartile ranges provided for the model configurations.Table 2 summarizes these estimates across models.
  • Sharded buffers: In sharded buffers, each trainer GPU maintains a separate replay buffer and samples only from its own stored trajectories.The reported total buffer size sums the sizes across trainer GPUs.
  • Positive-bias sampling: Positive-bias sampling retains the freshest (1 − δ)N rollouts together with the freshest δN correct rollouts outside that recent set.This strategy changes buffer composition by preferentially preserving correct trajectories.

E Additional Experimental Results

Additional experiments examine learning-rate choices, wall-time and compute efficiency, off-policiness, training dynamics, generalization across models and tasks, and sampling variants.

  • Learning-rate ablations identify 3.37 · 10−7 for Qwen3-0.6B and 6.810−8 for Qwen2.5-7B as the best speed–stability balances.
  • Wall-time experiments compare no-buffer training with multiple replay-buffer configurations for Qwen3-0.6B and Qwen2.5-7B.Figures 10 and 11 report median accuracy and interquartile ranges over at least four seeds per curve.
  • Off-policiness experiments vary rollout-worker update frequency without a buffer and track median accuracy with interquartile ranges across training steps.Curves are labeled by median off-policiness over the rollouts used, with at least four seeds per curve.
  • Qwen3-0.6B experiments analyze compute, training dynamics, best accuracy, and compute cost across buffer sizes and inference-worker/trainer ratios.Figures 13 and 14 report test accuracy, training accuracy, entropy, best accuracy, and compute needed to reach 98% of the best score.
  • The methods are tested on Qwen3-8B Lean coding tasks and Llama 3.2 3B OpenR1-Math-220k, with additional Qwen2.5-7B buffer configurations.Figures 16 and 17 report accuracy versus compute, while Figure 15 adds Qwen2.5-7B configurations for two worker/trainer ratios.
  • Sampling without replacement, including prioritizing unused samples, does not produce a clear impact relative to the standard uniform buffer implementation.The comparison includes vanilla sampling, uniform sampling without replacement, and a variant prioritizing never-used samples.
Loading 2604.08706v1…