Source-linked AI summary
Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
Xu Wan, Yansheng Wang, Wenqi Huang, Mingyang Sun
TL;DR
On-policy RLVR can waste experience and yield homogeneous rewards, limiting learning efficiency on difficult LLM post-training samples, while naive off-policy reuse risks instability. BAPO adapts historical-sample use through difficulty-aware batch construction, and the paper reports consistent baseline gains, including a 12.5% average accuracy improvement.
Problem
On-policy RLVR can waste experience and produce homogeneous rewards, limiting learning efficiency on difficult LLM post-training samples.
Source: §1 · p. 1Method
BAPO delays the rollout policy and constructs training batches by re-evaluating difficult historical samples and reusing high-quality trajectories.
Source: §1 · p. 2, §6 · p. 10Results
Across three reasoning tasks, BAPO outperforms baselines in convergence and training efficiency, with a 12.5% average accuracy improvement over baselines.
Source: §5.1 · p. 7, §6 · p. 10Takeaways & Limitations
Under certain assumptions, adaptive batch construction with KL-constrained updates mitigates the homogeneous-reward issue.
Source: §1 · p. 2The policy-improvement bound assumes rewards in [0, 1] and specified total-variation distance constraints.
Source: §3.3 · p. 5
Abstract
from arXiv · showhide
Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.
1 INTRODUCTION
The introduction frames on-policy RLVR as vulnerable to homogeneous rewards and wasted experience, while naive off-policy reuse can destabilize LLM post-training. It presents BAPO as an adaptive replay approach and reports gains over existing frameworks on difficult samples.
1 INTRODUCTION
- Homogeneous rewards can leave difficult samples with negligible effective gradients, while use-once on-policy rollouts discard valuable experience.Group-based advantage estimation depends on reward diversity; when rewards are identical, its policy-improvement lower bound collapses.Source: §1 · pp. 1–2
- Naively reusing off-policy samples may cause entropy collapse and performance degradation during LLM post-training.The introduction notes that off-policy reuse can improve sample efficiency and batch diversity in traditional RL, but may destabilize LLM training.Source: §1 · p. 2
- BAPO re-evaluates historical hard prompts to drive exploration and reuses high-quality trajectories through a dynamic quality threshold.This difficulty-aware replay strategy differs from simply mixing buffer and online data.Source: §1 · p. 2
- Under certain assumptions, BAPO’s adaptive batch construction and KL-constrained updates mitigate homogeneous rewards.The paper also reports that the method improves convergence and difficult-sample solving compared with existing frameworks.Source: §1 · p. 2
- BAPO achieves better convergence and greater improvement on difficult samples than existing on-policy and off-policy RLVR frameworks.The paper evaluates the method across multiple reasoning tasks and LLM backbones.Source: §1 · p. 2
2 RELATED WORK
The related-work section contrasts on-policy RLVR’s group-based updates and use-once sampling with off-policy methods that reuse historical or external experience. It also highlights policy-distribution mismatch and premature convergence as risks of replay.
2.1 ON-POLICY RL POST-TRAINING FRAMEWORK
- On-policy RLVR optimizes an LLM policy for deterministically verified outcome rewards, with GRPO estimating advantages from reward statistics within response groups.GRPO-style objectives constrain policy updates and divergence from a reference policy.Source: §2.1 · p. 2
- DAPO changes clipping and sampling to keep advantages nonzero, while GSPO uses sequence-level ratios; both remain on-policy and use each rollout once.DAPO is reported to consume approximately four times as many rollouts as GRPO; GSPO is described as more stable in MoE architectures.Source: §2.1 · pp. 2–3
2.2 OFF-POLICY RL POST-TRAINING FRAMEWORK
- Off-policy RL separates response generation from training-batch construction, drawing on past policies or external guidance for rollouts and replay buffers for training.Examples include asynchronous or delayed-policy rollouts and buffer strategies that select, mix, or refine historical samples.Source: §2.2 · p. 3
- Historical samples from different policy distributions can add noise and instability, while high-accuracy replay may suppress exploration and promote premature convergence.The section cautions that simply reusing past samples can hinder policy improvement.Source: §2.2 · p. 3
3 METHOD
BAPO combines delayed online rollouts with historical replay, constructing each training batch to retain reward variation while using difficult and high-quality samples selectively. Its analysis provides a lower bound on policy improvement for the adaptive batch under stated assumptions.
3.1 FORMAL DEFINITIONS
- BAPO’s objective combines filtered samples from a delayed rollout policy with historical buffer samples, using importance-sampling ratios for both sources.The buffer stores the probabilities of the policies that generated its responses for reuse.Source: §3.1 · p. 4
3.2 ADAPTIVE TRAINING BATCH CONSTRUCTION
- Adaptive batch construction selects filtered fresh samples, re-evaluated difficult prompts showing improvement, and recent high-quality buffer samples to fill remaining capacity.The high-quality threshold shifts from easier to harder instances according to global average performance.Source: §3.2 · pp. 4–5
- To limit instability from stale data, BAPO restricts its auxiliary high-quality buffer to trajectories from the three most recent steps.The buffer is sampled randomly to fill the batch capacity left by the other subsets.Source: §3.2 · p. 5
3.3 THEORETICAL ANALYSIS
- Under stated assumptions, BAPO’s adaptive batches yield a lower bound on expected policy improvement, with finite positive constants supporting bounded training stability.The theorem assumes bounded rewards and constraints on policy-distribution distances.Source: §3.3 · pp. 5–6
4 EXPERIMENTAL SETUP
The experiments evaluate on-policy and off-policy RLVR methods across mathematics, planning, and visual geometry tasks using several model backbones. They use shared comparative settings on eight A100 GPUs.
Task selection
- Experiments cover mathematics, Countdown planning, and visual geometry, using task-specific datasets, evaluation splits, and model backbones.Mathematics uses DeepSeek R1 Distilled 1.5B and Qwen3 8B; planning uses Qwen2.5 Math 1.5B and 7B; visual geometry uses Qwen2.5 VL 3B and 7B.Source: §4 · p. 6
Baselines
- Baselines include on-policy GRPO, DAPO, and MoPPS, plus off-policy GRPO with v = 5, RePO, and Remix-GRPO.The off-policy comparators use delayed rollouts or replay-buffer retrieval strategies.Source: §4 · p. 6
Implementation Details
- Comparative experiments use identical parameters and run on eight A100 GPUs with 80GB memory using the Verl framework.Further implementation details are given in Appendix A.7.Source: §4 · p. 7
5 RESULTS ANALYSIS
BAPO outperforms baselines in accuracy and convergence, while its batch-reconstruction components contribute to performance and its training efficiency remains competitive. It also improves initially unsolvable samples more often than GRPO.
5.1 MAIN RESULTS
- 12.5% average accuracy improvement over baselines, with BAPO also achieving smoother convergence and higher reward bounds under the same settings.Table 1 reports accuracy averaged over 32 runs; GRPO shows early training oscillations on mathematics.Source: §5.1 · p. 7, Table 1 · p. 7, Figure 4 · p. 7
5.3 DETAILED ANALYSIS
- DAPO approaches BAPO on some metrics but requires approximately 2.5× more rollouts, while BAPO maintains training speed comparable to GRPO.BAPO’s smaller assembled batches offset the overhead of off-policy re-evaluation and log-probability recomputation.Source: §5.1 · p. 7, §5.3 · pp. 9–10, Figure 9 · p. 10
5.2 MECHANISM ANALYSIS
- BAPO retains a clear advantage over GRPO in a parameter-free mini-test, supporting the role of its X2 and X3 batch structure rather than threshold tuning.The mini-test replays historical all-wrong samples as X2 and reuses exactly 50%-accuracy samples as X3.Source: §5.2 · p. 8
- Removing re-evaluated difficult samples X2 causes an approximately 21% performance drop, indicating that this component contributes materially to BAPO’s results.The ablation evaluates X2 and reused high-quality samples X3.Source: §5.2 · p. 8, Figure 6 · p. 9
- Performance remains stable across reasonable rollout-delay and re-rollout-frequency settings, while adaptive difficulty thresholds yield the best convergence and fixed thresholds still outperform baselines.Extreme delays degrade performance when policy divergence becomes excessive; the authors report that diverse historical data matters more than precise threshold values.Source: §5.2 · pp. 8–9, Figure 6 · p. 9
5.3 DETAILED ANALYSIS
- After 3 epochs, BAPO improves 31% of initially unsolvable samples, compared with 19% for GRPO.The analysis tracks samples that began at 0/8 accuracy in the DeepScalerR training subset.Source: §5.3 · p. 9, Figure 7 · p. 9
6 CONCLUSION
BAPO is an off-policy RLVR framework for LLM post-training that adapts batches by re-evaluating difficult samples and reusing historical high-quality ones. Across three reasoning tasks and different LLM backbones, it outperforms baselines in convergence and training efficiency; adapting it to MoE models and agentic RL remains a challenge.
6 CONCLUSION
- BAPO combines delayed rollout policies with re-evaluation of difficult samples and reuse of historical high-quality samples for LLM post-training.The conclusion reports validation on three reasoning tasks using different LLM backbones.Source: §6 · p. 10
ETHICS STATEMENT
The authors report adherence to the ICLR code of ethics and state that the study raises no privacy or security concerns.
ETHICS STATEMENT
- The authors report that the study raises no privacy or security concerns and follows established ethical guidelines for its procedures and data handling.They also state that they considered discrimination, bias, and fairness.Source: ETHICS STATEMENT · p. 11
REPRODUCIBILITY STATEMENT
The authors state that implementation details, experimental settings, hyperparameters, dataset specifications, and theoretical proofs are documented in supplementary materials or the appendix.
REPRODUCIBILITY STATEMENT
- The authors report that supplementary materials document implementation details and experimental settings, while the appendix contains theoretical proofs and assumptions.They state code and data will be made available upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 11
THE USE OF LARGE LANGUAGE MODELS
The authors say LLMs were used only to edit the manuscript’s language, not to contribute to its research or technical content.
THE USE OF LARGE LANGUAGE MODELS
- LLMs were used for grammar checking, style refinement, and language polishing; the authors state that the research ideas, experiments, analysis, and conclusions are their own.The authors say the tools did not contribute to the study’s conceptual or technical content.Source: THE USE OF LARGE LANGUAGE MODELS · p. 11
A.1 GLOSSARY OF TERMS AND NOTATIONS
The glossary defines notation for the policy, reward, replay buffer, sample groups, and training objective used in BAPO.
A.1 GLOSSARY OF TERMS AND NOTATIONS
- BAPO’s batch comprises fresh samples X1, re-evaluated difficult samples X2, and historical high-quality samples X3.The glossary also defines thresholds c1, c2, and c3 for classifying historical samples by difficulty using group mean reward.Source: §A.1 · p. 15
A.2 THEORETICAL ANALYSIS
The appendix analyzes BAPO’s policy-improvement bound and reports ablations, training behavior, computation, visualizations, settings, algorithm details, and a PPO extension.
A.2 THEORETICAL ANALYSIS
- BAPO’s expected improvement over filtered samples is lower-bounded under bounded rewards, with finite constants when c1, c2, and c3 lie in (0, 1).For binary rewards, the lower-bound contribution is maximized at mean group reward µ = 0.5, where reward variance is highest.Source: §A.2 · pp. 15, 17, 18
A.3 ONLINE FILTER MECHANISM ANALYSIS
- Range filtering achieves the highest asymptotic performance, Gaussian filtering converges faster initially, and uniform filtering collapses after 150 steps.The Gaussian filter prioritizes accuracy near 0.5, while the range filter retains a broader spectrum of valid samples; Figure 10 compares the filters’ gradient norms and Mean@8.Source: §A.3 · pp. 18–19, Figure 10 · p. 19
A.4 TRAINING DYNAMICS AND TEST CURVES
- BAPO and DAPO outperform GRPO in training rewards; BAPO also has higher entropy and longer responses, while test curves cover planning and visual geometry.Figures 11 and 12 show training dynamics and test curves.Source: §A.4 · p. 19, Figure 11 · p. 19, Figure 12 · p. 20
A.5 COMPUTATION ANALYSIS
- For mathematics, re-evaluation increases training time beyond GRPO but improves bad-case handling and overall test performance; removing re-evaluation speeds planning at a performance cost.The authors note that lower re-evaluation frequencies remain to be explored; Table 2 defines its time and batch-size measures.Source: §A.5 · p. 20, Table 2 · p. 20
A.6 VISUALIZATION
- BAPO’s training batches draw approximately 40-60% of actual training samples from online samples X1, with the remainder from X2 or X3; high-quality reasoning paths learned by RL are robust, with significant performance degradation negligible.Source: §A.6 · p. 22
A.7 HYPERPARAMETER SETTING
- The mathematics configuration specifies group size 8, a maximum response length of 8192, learning rate 1e-6, and three training epochs.Some parameters differ for planning and visual geometry tasks.Source: Table 3 · p. 22
A.8 ALGORITHM
- Algorithm 1 integrates BAPO with GRPO-like RLVR by combining filtered fresh samples, periodically re-evaluated difficult samples, and sampled historical high-quality data.The resulting batch is used to compute advantages and update the actor and critic.Source: §A.8 · p. 23
A.9 GENERALIZATION ANALYSIS
- +5.5 on AIME 2024: BA-PPO outperformed standard PPO with Qwen3-4B, supporting the batch-adaptation approach beyond GRPO.The experiment used Qwen3-4B actor and critic networks trained on DeepScaleR.Source: §A.9 · p. 24, Figure 16 · p. 24