Source-linked AI summary
BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping
Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, Xun Deng, Zhikai Lei, Miao Zheng, Guoteng Wang, Shuo Zhang, Peng Sun, Rui Zheng, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
Off-policy RL for LLMs is sample-efficient but can suffer unstable optimization, entropy collapse, and over-exploitation. BAPO analyzes these mechanisms and adaptively changes clipping bounds to rebalance updates and preserve entropy. Across off-policy settings, it achieves strong AIME results, with 32B models outperforming comparable open-source and leading proprietary systems.
Problem
Off-policy RL improves sample efficiency but can exhibit unstable optimization, sharply declining entropy, gradient explosions, and collapse as data becomes stale.
Method
BAPO dynamically adjusts clipping bounds to rebalance positive and negative token contributions, incorporate low-probability positives, filter excessive negatives, and preserve policy entropy.
Results
Across diverse off-policy settings, 32B BAPO reaches 87.1 on AIME24 and 80.0 on AIME25, outperforming comparable open-source and leading proprietary systems.
Takeaways & Limitations
BAPO provides stable optimization and competitive performance across model scales, backbones, and off-policy training settings.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has recently become the core paradigm for aligning and strengthening large language models (LLMs). Yet, applying RL in off-policy settings--where stale data from past policies are used for training--improves sample efficiency, but remains challenging: policy entropy declines sharply, optimization often becomes unstable and may even collapse. Through theoretical and empirical analysis, we identify two key insights: (i) an imbalance in optimization, where negative-advantage samples dominate the policy gradient, suppressing useful behaviors and risking gradient explosions; and (ii) the derived Entropy-Clip Rule, which reveals that the fixed clipping mechanism in PPO-like objectives systematically blocks entropy-increasing updates, thereby driving the policy toward over-exploitation at the expense of exploration. Building on these insights, we propose BAlanced Policy Optimization with Adaptive Clipping (BAPO), a simple yet effective method that dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization. Across diverse off-policy scenarios--including sample replay and partial rollout--BAPO achieves fast, stable, and data-efficient training. On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B, while our 32B BAPO model not only achieves state-of-the-art results among models of the same scale but also outperforms leading proprietary systems like o3-mini and Gemini-2.5-Flash-Thinking.
1. Introduction
Off-policy RL improves sample efficiency but becomes unstable as data staleness increases, with declining entropy and possible collapse. BAPO addresses these problems by analyzing optimization imbalance and clipping-induced entropy loss, then adaptively balancing policy updates.
- Motivation: Increasing data staleness causes unstable optimization, exploding gradients, sharply declining entropy, and even training collapse, unlike stable on-policy training.The off-policy setting offers sample-efficiency advantages but introduces these optimization challenges.
- Motivation: The analysis identifies negative-advantage dominance and fixed PPO-like clipping as mechanisms that suppress useful updates and drive policies toward over-exploitation.The clipping mechanism blocks entropy-increasing updates while over-penalizing low-probability negatives.
- BAPO: BAPO dynamically adjusts clipping bounds to rebalance positive and negative contributions, incorporate low-probability positives, filter excessive negatives, and preserve policy entropy.The method targets a better exploration–exploitation balance in off-policy RL.
- Results: 7B BAPO scores 70.8 on AIME24 and 62.5 on AIME25, surpassing open-source counterparts such as SkyWork-OR1-7B.Experiments span sample replay, partial rollout, and varying staleness.
- Results: 32B BAPO reaches 87.1 on AIME24 and 80.0 on AIME25, outperforming Qwen3-32B and proprietary systems including o3-mini-medium and Gemini-2.5-Flash-Thinking.These are the paper’s reported headline benchmark results.
- Validation: The paper validates BAPO across multiple backbones, model scales, and off-policy settings, reporting stable optimization and competitive results with proprietary systems.The contribution summary emphasizes broad experimental validation.
2. Preliminaries
Policy-gradient RL trains an LLM by maximizing expected reward through estimated advantages. PPO-like objectives use importance sampling to correct behavior–target policy mismatch while clipping ratios to limit destabilizing updates.
- Policy-Gradient RL: Given prompts and rewards, policy-gradient methods optimize an LLM by maximizing expected reward through gradient ascent.The policy generates a token sequence, and the objective is optimized over the training dataset.
- Policy-Gradient RL: The advantage A_t measures how much better token action y_t is than the expected action under the current policy.Advantages provide the learning signal used in the policy gradient.
- PPO Objective: PPO-like training optimizes a surrogate objective that takes the minimum of the unclipped and clipped importance-weighted advantage terms.The displayed objective applies clipping around the importance ratio.
- Importance Sampling and PPO Objective: The importance weight r_t corrects distribution mismatch between rollout and target policies, while clipping implicitly enforces a trust region against overly large updates.The clipping interval width is controlled by ε ∈ (0, 1).
- Policy-Gradient Decomposition: The policy gradient is analyzed separately for positive- and negative-advantage tokens to expose their distinct contributions.This decomposition motivates the later analysis of optimization imbalance.
3. Motivation: Imbalanced Optimization and Entropy-Clip Rule
Off-policy staleness produces instability and entropy loss because negative samples can dominate optimization while fixed clipping blocks entropy-increasing positive updates. The Entropy-Clip Rule motivates selectively widening clipping for positive tokens and restricting excessive negatives.
- Preliminary Results: Increasing data staleness makes off-policy training less stable, lowers entropy more severely, and clips more tokens than on-policy training.The observed effects reflect reduced exploratory capacity.
- Imbalanced Optimization: Positive samples are a minority in both token count and policy-gradient contribution, while longer trajectories on difficult queries produce more negative tokens.Figure 6 connects longer responses with the excess of negative samples.
- Imbalanced Optimization: Merely enlarging the upper clipping bound amplifies positive signals but does not prevent negative data from dominating optimization.This limitation motivates balancing both clipping bounds.
- Entropy-Clip Rule: The Entropy-Clip Rule states that entropy changes are driven by unclipped tokens through the covariance between their log probabilities and advantages.The rule is derived for the PPO surrogate objective.
- Entropy-Clip Rule: Symmetric clipping excludes many low-probability positive tokens, suppressing entropy-increasing updates and causing entropy decline and reduced exploration.Tokens with very high or very low importance-sampling weights tend to have low probabilities.
- Summary of Motivation: The motivation is to balance positive and negative contributions, prevent gradient explosion, and preserve entropy for continued exploration.These goals summarize the paper’s two central optimization concerns.
4. Methodology
BAPO stabilizes off-policy RL by asymmetrically and adaptively adjusting clipping bounds to increase useful positive contributions while limiting harmful negative ones.
- 4.1. Validation Experiment: Asymmetric Clipping: BAPO adjusts c_low and c_high asymmetrically because relaxing the lower bound accelerates entropy collapse, whereas increasing the upper bound improves performance and counteracts entropy decline.The asymmetric-clipping experiments motivate adaptive rather than fixed clipping.
- 4.2. BAPO: BAlanced Policy Optimization with Adaptive Clipping: BAPO dynamically increases c_high and c_low until the positive-token contribution reaches a target threshold ρ_0.The bounds change in steps δ_1 and δ_2 within prescribed movable ranges.
- 4.2. BAPO: BAlanced Policy Optimization with Adaptive Clipping: By increasing positive-token contributions while preventing negative tokens from dominating, BAPO preserves entropy and balances exploration with exploitation.The target positive contribution also avoids uncontrolled entropy growth and tail degradation.
- 4.2. BAPO: BAlanced Policy Optimization with Adaptive Clipping: The method samples and filters responses, computes rewards and advantages, then updates the policy using dynamically adjusted clipping bounds.This sequence is summarized by the algorithmic procedure.
22 end
BAPO’s training dynamics show stable rewards, gradients, and entropy, while adaptive clipping responds to staleness and avoids manual bound tuning.
- Stable and fast training of BAPO.: BAPO produces rapidly increasing rewards, greater positive-token contributions, steady gradient normalization, and stable policy entropy.These dynamics correspond to a better exploration–exploitation balance.
- Stable and fast training of BAPO.: The averaged upper and lower clipping bounds fluctuate during training, confirming that BAPO adapts both bounds rather than using fixed values.This adaptive behavior eliminates complex manual hyperparameter tuning compared with DAPO and asymmetric clipping.
- Stable and fast training of BAPO.: Under different data-staleness levels, BAPO consistently outperforms both the baseline and Clip-Higher approach.The comparison uses R1-Distill on the SkyWork-OR1-RL dataset.
- Stable and fast training of BAPO.: As importance-sampling weights deviate further from 1, token probabilities decrease and low-probability tokens often exhibit higher entropy.This relationship motivates BAPO’s treatment of low-probability positive and negative tokens.
5. Experiments and Discussion
Across model scales, backbones, and off-policy settings, BAPO improves benchmark performance and maintains stable training, including under partial rollout.
- 5. Experiments and Discussion: The experiments evaluate AIME24 and AIME25 across DeepSeek-R1-Distill, OctoThinker, and BP-Math backbones using replay, partial rollouts, and varying staleness.The study compares SFT, GRPO, commercial models, and open-source baselines.
- 5.2. Main Results: BAPO improves over SFT by 2.7 and 1.9 points on AIME24 and AIME25 with BP-Math-32B, and by 3.9 and 3.5 points with BP-Math-7B.GRPO provides only 0.2 and 0.7 points on the BP-Math-32B model.
- 5.2. Main Results: BP-Math-32B BAPO surpasses Qwen3-32B by 5.7 and 7.1 points on AIME24 and AIME25, respectively, and SkyWork-OR1-32B by 4.9 and 6.7 points.Among 7B models, it improves AIME25 by 7.9 points over SkyWork-OR1-7B.
- 5.2. Main Results: BP-Math-32B BAPO surpasses DeepSeek-R1 by 7.3 and 10.0 points on AIME24 and AIME25, respectively, while performing comparably to o3-mini.BP-Math-7B BAPO performs on par with Gemini-2.0-Flash-Thinking.
- 5.3. Generalization: BAPO achieves competitive results and greater training stability on Llama-based models.These results are reported in Table 2 and Figure 13.
6. Related Work
Related work frames PPO and GRPO as mainstream LLM RL methods and describes subsequent efforts to improve optimization stability, efficiency, and balance.
- 6. Related Work: PPO constrains policy updates with a clipping-based surrogate objective, while GRPO supports long-horizon reasoning.Both are presented as mainstream policy-gradient methods for LLM reinforcement learning.
- 6. Related Work: Recent methods target RL optimization stability and performance ceilings through techniques such as Clip-Higher and dynamic sampling.The paper positions these efforts within broader attempts to balance LLM RL optimization.
7. Conclusion
The paper analyzes off-policy RL instability in LLMs and proposes BAPO to balance policy updates, preserve exploration, and stabilize training.
- BAPO dynamically adjusts clipping bounds to balance positive and negative samples while preserving exploratory capability during training.
A. Performance on OctoThinker-Llama
The OctoThinker-Llama experiments report consistent BAPO improvements, with training settings including adaptive clipping bounds and a target positive loss contribution.
- BAPO provides consistent and significant improvement in OctoThinker-Llama training.The experiment uses a model selected after mid-training to demonstrate robustness.
- The training configuration sets the low bound to 0.8-0.9, the high bound to 1.2-2.0, and the target positive loss contribution to 0.45.
- The analysis derives PPO entropy behavior using tabular softmax assumptions and relationships between policy updates, clipping, and token contributions.
B.4.1. Direct Analysis: Why Varying 𝜀Alters Entropy?
The direct analysis shows that clipping determines which tokens contribute to entropy, so changing the clipping parameter changes the number of entropy-contributing tokens.
- The analysis examines the covariance contribution of a clipped token to characterize its effect on entropy.
- Only retained tokens contribute to the overall entropy.
- Altering the parameter 𝜀 manipulates the number of tokens that can contribute to entropy.
B.4.2. Advanced Analysis : Which Type of Tokens Matter Most for Entropy?
The advanced analysis classifies tokens by probability and advantage, then identifies which combinations decrease or increase policy entropy.
- The analysis experiment uses naive REINFORCE without clipping, classifying high and low advantages by the sign of their advantage values.
- The analysis defines token categories using probability and advantage characteristics before deriving their entropy effects.
- A token decreases entropy when ΔH(𝑦𝑠) < 0 and increases it otherwise.
- High-probability, high-advantage positive tokens and low-probability, low-advantage negative tokens reduce overall entropy.
- Positive high-probability tokens with low advantages and negative high-probability tokens with low advantages increase overall entropy.