BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

Yuan Li; Bo Wang; Yufei Gao; Yuqian Yao; Xinyuan Wang; Zhangyue Yin; Xipeng Qiu

Source-linked AI summary

BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning

Yuan Li, Bo Wang, Yufei Gao, Yuqian Yao, Xinyuan Wang, Zhangyue Yin, Xipeng Qiu

arXiv:2603.04918v1Mar 5, 2026cs.LGcs.AI

TL;DR

Fixed clipping bounds can suppress exploration by low-probability, high-advantage actions, limiting the alignment between proximal stability and effective exploration. BandPO projects f-divergence trust regions into dynamic, probability-aware clipping intervals and consistently outperforms clipping baselines while mitigating entropy collapse.

Problem
Fixed clipping bounds can prematurely suppress low-probability, positive-advantage actions, limiting the reconciliation of proximal constraints with exploration.
Method
BandPO projects f-divergence trust regions into dynamic, probability-aware clipping intervals through its unified Band operator.
Results
Across models from 1.5B to 8B, BandPO outperforms heuristic clipping baselines and robustly prevents entropy collapse.
Takeaways & Limitations
BandPO provides geometrically grounded proximal constraints that resolve the exploration bottleneck for tail strategies while maintaining simplex consistency.
Takeaways & Limitations
BandPO with KL divergence requires iterative root-finding, adding computational latency and minor approximation errors relative to elementary scalar operations.

Abstract

from arXiv · show

Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.

1 Introduction

The paper identifies fixed ratio clipping as an exploration bottleneck because it suppresses upward updates for low-probability, high-advantage actions and contributes to entropy reduction. BandPO addresses this by projecting f-divergence trust regions into dynamic, probability-aware clipping intervals, improving performance while mitigating entropy collapse.

Motivation: RLHF uses proximal policy-update constraints to balance optimization stability and exploration, with PPO emulating trust-region updates through surrogate-objective clipping.This avoids the expensive Fisher Information computations required by TRPO.
Problem: Canonical clipping biases policy entropy downward and silences gradients for low-probability, high-advantage actions, preventing reinforcement of novel tail strategies.With symmetric thresholds, entropy-decreasing upper clipping dominates entropy-increasing lower clipping even under purely random rewards.
Problem: Fixed ratio bounds make the maximum feasible probability variation scale linearly with old probability, causing vanishing upward margins for low-probability positive-advantage actions.These actions therefore become susceptible to premature clipping, restricting effective exploration.
Method: BandPO introduces Band, a unified operator that projects f-divergence-induced trust regions into dynamic, probability-aware clipping intervals.The method replaces GRPO clipping with Band and uses a single interpretable radius parameter.
Method: BandPO’s projected bounds expand the feasible upward margin for low-probability actions, preventing premature clipping and preserving critical exploration gradients.The KL example illustrates this adaptive expansion through the highlighted blue region.
Results: BandPO consistently improves performance over GRPO and GRPO with clip-higher across Qwen2.5 (3B, 7B) and Llama3 (8B) on multiple mathematical benchmarks while mitigating entropy collapse.The introduction reports these empirical gains across the listed model families and benchmarks.

2 Related Work

Related work progresses from KL-based trust regions to ratio clipping and adaptive clipping variants, including methods designed for LLM reinforcement learning. The paper identifies limited theoretical guidance for clipping bounds and introduces BandPO as a probability-aware bridge between f-divergence trust regions and clipping intervals.

From Trust Regions to Ratio Clipping: Trust-region methods constrain updated policies to remain near sampling policies, with Schulman et al. introducing KL-induced neighborhoods for policy optimization.
Adaptive Clipping Variants: Adaptive clipping variants address bound selection and instability through auxiliary lower bounds, state-wise clipping, and other simplified adaptive mechanisms.
Clip Control in LLM: In LLM reinforcement learning, DAPO’s Clip-Higher decouples and relaxes upper bounds, while DCPO derives dynamic clipping boundaries through inequality relaxation.
Clip Control in LLM: Existing LLM reinforcement learning clipping methods lack a principled, interpretable framework for controlling bounds while balancing proximal constraints and exploration.
Clip Control in LLM: BandPO projects f-divergence-induced trust regions into probability-aware clipping intervals using an interpretable parameter.

3 Preliminaries

The paper formulates LLM alignment as a discrete Markov Decision Process in which an autoregressive policy generates token sequences and maximizes expected sequence-level reward. It then introduces the probability-ratio and clipping framework underlying GRPO, including token-averaged objectives, group-standardized advantages, KL regularization, and potentially asymmetric bounds.

LLM Alignment as an MDP: LLM alignment is modeled as a discrete Markov Decision Process with token actions, autoregressive responses, and states formed from the prompt and preceding tokens.The policy maps each state to a conditional action distribution over a vocabulary of size V.
LLM Alignment as an MDP: The optimization objective maximizes expected reward over prompts and policy-generated responses, using a sparse sequence-level scalar reward typically derived from verification signals.The objective is J(θ) = E_x∼D,y∼πθ[R(x,y)].
Policy Optimization Preliminaries: Iterative updates use the probability ratio between the new and old policies as an importance-sampling weight, with clipping imposing proximal constraints.GRPO inherits this clipping mechanism while avoiding reliance on a critic model.
Policy Optimization Preliminaries: The GRPO objective averages over tokens, standardizes sequence rewards within groups into advantages, applies KL regularization toward a reference policy, and permits asymmetric clipping bounds.The generalized formulation allows ε+ > ε−, whereas standard GRPO uses symmetric bounds with ε+ = ε−.

4 The Bottleneck in Canonical Clipping

Canonical clipping uses fixed ratio bounds that make allowable probability updates scale with the old action probability. This suppresses exploration for low-probability, positively advantaged actions, while loosening the bound enough to help them makes it ineffective for high-probability actions.

Canonical clipping confines r_t(θ) to the fixed interval [1 − ϵ−, 1 + ϵ+], with ϵ−, ϵ+ > 0 fixed constants.
Fixed ratio bounds constrain probability variations to scale linearly with π_old(a|s), so feasible upward shifts vanish as action probability approaches zero.This causes premature clipping for low-probability actions with positive advantages and effectively nullifies their gradient contributions.
For π_old = 0.08 and ϵ+ = 0.2, the allowed update Δπ≤0.016 is negligible relative to the theoretical capacity (0.92).
Enabling Δπ≈0.4 for the same tail token would require raising ϵ+ to ≈5.0, permitting Δπ≤4.0 for π_old = 0.8 versus its physical limit of 0.2.Thus, static clipping cannot simultaneously support meaningful tail-token exploration and maintain effective constraints for head tokens.

5 Method

BandPO introduces the Band operator, which projects f-divergence trust regions into probability-aware ratio intervals, replacing fixed canonical clipping. Its convex optimization formulation admits globally optimal solutions and an exact scalar reduction, while its bounds expand for low-probability actions to support exploration.

Optimal Dynamic Bounds: The upper and lower ratio bounds are obtained by convex programs whose local optima are globally optimal under strictly convex 𝑓 and 𝑃(𝑎) ∈(0, 1).The upper bound maximizes 𝑄(𝑎), while the lower bound is obtained symmetrically.
Band Operator: Band projects f-divergence trust regions onto action-specific probability-ratio intervals, replacing the fixed clipping interval [1 − 𝜖, 1 + 𝜖].The resulting bounds are governed by the interpretable trust-region radius 𝛿.
Exact Scalarization: Uniform complement rescaling reduces the high-dimensional simplex optimization to a single target-action ratio 𝑟, yielding a univariate divergence constraint and root-finding problem.The complement scaling factor is uniquely determined by simplex normalization, and the two roots provide the projection clipping bounds.
Exploration Bottleneck: As 𝑝→0+, the upper bound approaches +∞ and the lower bound approaches 0, while as 𝑝→1− the upper bound approaches 1.These asymptotic behaviors theoretically address the exploration bottleneck by avoiding restrictive bounds on low-probability actions.
Exploration Bottleneck: For 𝛿> 0, the upper bound strictly decreases with probability 𝑝, whereas the lower bound strictly increases with 𝑝.BandPO and DCPO therefore expand ratio bounds as 𝑝→0, while BandPO maintains consistency with the simplex constraints.
BandPO Framework: BandPO substitutes GRPO’s canonical clipping mechanism with the theoretically rigorous Band operator.The framework combines optimization stability with effective exploration of tail strategies.

6 Empirical Study

BandPO is evaluated across multiple model scales, datasets, and clipping baselines, consistently improving mean@32 while mitigating instability. Training-dynamics analyses show that probability-aware bounds suppress erroneous tail-action clipping and prevent entropy collapse without simply reducing aggregate clipping.

Models and Datasets: Experiments fine-tune four models on DAPO and MATH Levels 3–5, evaluating reasoning robustness on AMC 2023, AIME 2024, and AIME 2025.The models include Qwen2.5-3B-Instruct and DeepSeek-R1-Distill variants at 1.5B, 7B, and 8B scales.
Main Results: BandPO consistently outperforms GRPO and Clip-Higher in mean@32, improving over GRPO by at least 2.0 points across all settings and by approximately 10 points on Qwen2.5-3B AMC2023.Clip-Higher regresses by 1–2 points on the 7B AIME benchmarks.
Hyperparameter Analysis: The KL trust-region radius δ=0.05 consistently delivers the strongest BandPO performance, with smaller models more sensitive to suboptimal settings than larger models.For smaller models, suboptimal settings reduce mean@32 by approximately 10% and pass@32 by 5%, while 7B fluctuations remain within 2–3%.
Training Dynamics: Band maintains an aggregate clip rate close to canonical clipping, whereas Clip-Higher reduces the overall clip rate by approximately 50%.This reflects Band’s tighter constraints in the high-probability regime compared with Clip-Higher.
Training Dynamics: Band reduces upper-bound clipping for low-probability actions to nearly zero and raises converged mean entropy from 0.02 to 0.2, preventing early entropy collapse.Fixed-bound methods attribute approximately 20% of total clipping to tail tokens, rising to 60% during canonical clipping’s first 50 steps.

7 Discussion

BandPO realigns proximal clipping with trust-region geometry by projecting f-divergence constraints into dynamic, probability-aware intervals, addressing tail-strategy exploration while preserving simplex consistency. The discussion highlights numerical-solver overhead and proposes adaptive Band operators driven by token-level uncertainty metrics.

Implications: BandPO projects f-divergence trust-region constraints into dynamic, probability-aware intervals, resolving the exploration bottleneck for tail strategies while maintaining strict simplex consistency.
Limitations: BandPO introduces numerical complexity because it solves convex constraint equations rather than using computationally trivial canonical clipping.Closed-form solutions for TV and Pearson χ2 retain analytical efficiency, whereas KL divergence requires iterative root-finding.
Future Work: Future work will develop adaptive Band operators whose radius δ_t is dynamically modulated by token-level metrics such as policy entropy or semantic uncertainty.The proposed approach would tighten constraints for low-entropy syntactic transitions and relax bounds for high-stakes reasoning steps.

8 Conclusion

BandPO bridges efficient clipping and rigorous trust-region constraints by projecting f-divergence balls into dynamic, probability-aware bounds. Experiments across 1.5B–8B models show improved performance over heuristic baselines while maintaining training stability and addressing exploration bottlenecks for high-advantage tail actions.

Conclusion: BandPO bridges computationally efficient clipping with rigorous trust-region constraints through a principled optimization framework.Its bounds are obtained by projecting f-divergence balls into dynamic, probability-aware intervals.
Conclusion: Dynamic, probability-aware bounds address the exploration bottleneck for high-advantage tail actions while maintaining training stability.
Conclusion: Across models from 1.5B to 8B, BandPO significantly outperforms heuristic baselines.The conclusion also reports that BandPO robustly prevents [truncated passage].

Appendix · A Omitted Proofs · A.1 Proof of Lemma 1

The proof characterizes the optimal policy under the divergence constraint: the target action is fixed, while the remaining probability mass is distributed proportionally to the reference policy. Strict convexity establishes this symmetric structure, and a small trust-region radius ensures the solution remains strictly interior.

A.1 Proof of Lemma 1: The proof analyzes the maximization formulation, with the minimization case following identically because both share the same feasible set.
A.1 Proof of Lemma 1: Fixing the target-action mass q, the remaining mass 1 − q is allocated over complement actions to minimize their divergence contribution.
A.1 Proof of Lemma 1: Strict convexity and symmetry force the likelihood ratio to be constant across all complement actions at the global optimum.
A.1 Proof of Lemma 1: Therefore, the optimal complement policy satisfies Q(b) = cP(b) for every action b ≠ a.
A.1 Proof of Lemma 1: The probability-mass constraint determines the scaling factor as c = (1 − q)/(1 − P(a)).
A.1 Proof of Lemma 1: Using q = rP(a), the scaling factor is recovered as a function of the target-action ratio, c(r) = 1 − rP(a).
A.1 Proof of Lemma 1: Because Softmax reference probabilities are positive and δ is small, the feasible optimum has q < 1 and therefore c > 0.
A.1 Proof of Lemma 1: Consequently, every complement probability satisfies Q(b) > 0, so the optimum lies strictly inside the probability simplex.

A.2 Proof of Theorem 1 … A.7 Proof of Proposition

The appendices establish that strict convexity makes the binding equation’s roots uniquely determine optimal bounds, characterize their asymptotic and monotonic behavior, and provide saturation-aware numerical procedures. They prove global bisection convergence, show KL requires numerical solution, and derive divergence-specific bounds for Total Variation and Pearson χ2.

A.2 Proof of Theorem 1; 2. Global Minimum. We examine the critical point where: Strict convexity of g_f(p, r) gives a unique global minimum at r = 1 and makes g_f(p, r) = δ have exactly one root on each side when feasible.The feasible set is the interval between the two roots, so optimizing the ratio places the solution on the binding boundary.
A.3 Proof of Proposition 1 (Asymptotic Behavior): As p→0+, the lower bound tends to 0 when lim p→0 p f(0) = 0 < δ, while the upper bound tends to +∞.The tail constraint asymptotically becomes p f(r) ≤ δ, forcing the upper solution outward as δ/p diverges.
A.3 Proof of Proposition 1 (Asymptotic Behavior): As p→1−, the upper bound converges to 1, whereas the lower bound converges to a divergence-dependent constant, including max(1 − δ, 0) for one case and e^−δ for KL.The upper limit is squeezed by the simplex domain r ≤ 1/p; the appendix states that these limits align with Figure 2.
A.4 Proof of Proposition 2 (Monotonicity); 1. Positivity of 𝜕𝐹: The implicit-function analysis establishes strict monotonicity of the lower and upper roots with respect to p through the signs of the binding equation’s partial derivatives.It uses strict convexity and positivity of the associated Bregman divergence away from r = 1.
A.5 Proof of Proposition 3 (Constraint Activity and Saturation): Constraint activity determines whether each bound saturates at a simplex boundary or is the unique interior root of g_f(p, r) = δ.For the upper side, inactivity yields r_f,δ(p) = r_max; for the lower side, inactivity yields r_f,δ(p) = 0.
A.6 Numerical Implementation Details; A.6.1 Convergence Guarantees; A.6.2 Standard Bisection Solver Algorithms: In the active regime, bisection on [r_min, 1] and [1, r_max] converges linearly to the two unique roots to arbitrary precision ε.The solver applies opposite monotonicity directions on the lower and upper intervals.
A.6.3 Instantiation: Forward KL Divergence: For forward KL, the scalarized binding equation has no elementary closed-form solution, so the generic numerical solver is required to compute exact Band bounds.The cancellation of linear terms leaves a transcendental equation involving r logarithmically inside and outside the complement scaling argument.