PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning

Renjie Mao, Xiangxin Zhou, Lvfang Tao, Yixin Ding, Yu Shi, Yongguang Lin, Yuheng Wu, Honglin Zhu, Qian Qiu, Wenxi Zhu

arXiv:2606.10968v2cs.LGcs.AI

TL;DR

Uniform token-level trust regions overlook autoregressive asymmetry and cumulative prefix drift in RLVR. CPPO addresses this with position-weighted thresholds and a cumulative prefix budget, achieving the best validation AIME24/25/26 Avg@16 across four Qwen3 settings.

  • Problem

    Uniform token-level thresholds overlook autoregressive asymmetry and cumulative prefix drift, allowing early deviations to affect longer suffixes and errors to accumulate within prefixes.

  • Method

    CPPO uses a decreasing position-weighted threshold and cumulative prefix budget to constrain token-level updates according to autoregressive generation order.

  • Results

    CPPO achieves the best validation AIME24/25/26 Avg@16 across four Qwen3 settings spanning dense, MoE, Base, and post-trained models.

  • Takeaways & Limitations

    CPPO provides a trust-region mechanism aligned with autoregressive structure, with ablations attributing gains to both the position weight and prefix budget.

  • Takeaways & Limitations

    Uniform token-level thresholds fail to account for downstream error propagation and prior prefix drift.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become standard for improving LLM reasoning. However, existing PPO-style trust-region mechanisms remain position-agnostic by enforcing uniform thresholds across all tokens independently. This pointwise treatment conflicts with autoregressive generation in two critical ways. First, uniform thresholds ignore autoregressive asymmetry. Early-stage deviations produce compounding sequence-level drift, causing static thresholds to under-regulate early divergence and excessively constrain late-stage exploration. Second, evaluating token-level divergence in isolation overlooks cumulative prefix drift, granting the same divergence allowance regardless of how far the conditioning history has already deviated from the rollout policy. To address this limitation, we propose CPPO (Cumulative Prefix-divergence Policy Optimization), a token-level masking rule that aligns updates with a finite-horizon policy-improvement bound via two coupled mechanisms. First, a position-weighted threshold imposes stricter limits at early positions whose effects persist longer, relaxing constraints for late-stage tokens. Second, a cumulative prefix budget tracks historical deviations, dynamically restricting further token-level deviation to prevent compounding errors along the prefix. Empirically, CPPO enhances training stability and significantly improves reasoning accuracy across various model scales.

1 Introduction

RLVR’s off-policy PPO/GRPO updates can drift from the rollout policy, with autoregressive early-token deviations amplifying instability and reasoning degradation. CPPO addresses this by combining position-weighted token constraints with a cumulative prefix budget, achieving the best AIME24/25/26 average scores across model sizes.

  • Motivation: Off-policy RLVR reuses responses sampled from a fixed rollout policy µ while optimizing π, so unconstrained updates can become unstable and degrade reasoning performance.Early token deviations further alter the conditioning of all subsequent generation steps.
  • Limitations: PPO and GRPO clip sampled-token likelihood ratios, but their uniform position-agnostic thresholds overlook autoregressive asymmetry and allow larger sequence-level shifts from early-token deviations.Early tokens condition the entire subsequent generation, so identical token-level divergence has greater impact at earlier positions.
  • CPPO: CPPO aligns token-level updates with autoregressive generation by controlling where and how much policy deviation may accumulate along a trajectory.It replaces isolated pointwise limits with two coupled constraints addressing structural mismatches in token-level trust regions.
  • CPPO: CPPO formalizes that early token-level policy shifts affect longer suffixes and contribute more to the finite-horizon error bound than later shifts.This explains why uniform divergence thresholds become loose for long responses.
  • Contributions and evaluation: CPPO uses a position-weighted token-level constraint and a cumulative prefix budget, integrates as a drop-in token mask, and obtains the best AIME24/25/26 average scores across varying model sizes.Ablations support both constraints under matched RLVR settings.

2 Preliminaries

LLM reinforcement learning is formulated as a finite-horizon autoregressive decision problem with verifier rewards, where off-policy updates optimize a target policy from responses sampled by a rollout policy. Existing token-level trust regions use uniform divergence thresholds, motivating CPPO’s position-weighted and cumulative-prefix constraints.

  • Problem setup: LLM reinforcement learning generates responses autoregressively, with each state comprising the prompt and prior tokens and each action selecting the next token.The response probability factorizes over conditional next-token distributions.
  • Problem setup: After generation, a verifier supplies a bounded scalar reward, while RLVR reuses responses from a fixed rollout policy µ to optimize target policy π under common support.The objective is J(π) = E_x,y∼π[R(x, y)] with |R(x, y)| ≤ ξ.
  • Trust-region methods: Trust-region methods constrain target-policy movement from the rollout policy, while PPO and GRPO implement heuristic token-level approximations of the constrained update.TRPO’s explicit divergence constraint guarantees monotonic improvement when π remains close to µ.
  • Limitations of uniform trust regions: Existing token-level divergence methods apply a static threshold δ across positions, although early deviations affect longer suffixes than late deviations.The asymmetry arises because dropped future likelihood-ratio corrections propagate each deviation through subsequent tokens.
  • Limitations of uniform trust regions: Uniform thresholds also ignore cumulative prefix drift, allowing locally bounded deviations to shift the historical context far from the rollout-policy distribution.The resulting divergence budget remains constant regardless of prior prefix deviation.
  • CPPO motivation: CPPO addresses these limitations with a position-weighted threshold that is tighter early and looser late, together with a cumulative prefix budget.Both mechanisms redistribute the permitted standard token-level divergence along the response while bounding accumulated deviation.

3 CPPO

CPPO improves the finite-horizon policy-improvement bound by combining position-weighted token limits with cumulative prefix constraints. Its masking rule enforces these constraints online, preventing early deviations from compounding while relaxing late-token limits.

  • Finite-horizon bound: The finite-horizon bound assigns token divergence a coefficient λ_t = 4ξ¯ℓ(T −t), formalizing stronger error propagation for early deviations.The coefficient grows linearly with the remaining horizon, capturing compounding covariate shift in autoregressive generation.
  • Position-weighted constraint: CPPO aligns position weights with this asymmetry, imposing stricter divergence limits early and naturally relaxing them as generation progresses.The weight is designed to mirror the remaining-horizon error-propagation profile.
  • Cumulative prefix constraint: CPPO constrains both weighted token divergence and every intermediate weighted prefix average: w_tD_t ≤ c_t and P_m ≤ δ_bW_m.The prefix constraint prevents the policy from saturating the divergence budget at consecutive steps and limits cumulative historical drift.
  • Policy-improvement bound: Under a uniform token-level threshold, CPPO replaces the pointwise divergence factor δ with the prefix-average threshold δ_b, improving the bound when δ_b < δ.This comparison attributes the improvement to preventing many early prefixes with large remaining-horizon coefficients from simultaneously reaching the token-level threshold.
  • Online masking rule: The implementation converts the cumulative requirement into a preceding-prefix-dependent effective threshold, taking the minimum of the token-level threshold and the prefix-adjusted threshold.This enforces the prefix constraint during training when only the prefix generated so far is available.
  • Online masking rule: The initial prefix slack is an implementation detail that changes the formal inequality by at most a lower-order O(T ¯ℓδ) term in the Abel bound.It is not part of the formal theoretical bound.

4 Experiments

Experiments evaluate CPPO under matched training conditions across four Qwen3 settings and find consistent gains over trust-region baselines. Results also show improvements remain robust across hyperparameter choices and divergence formulations.

  • Experimental Setup: Experiments use DAPO-Math-17k with the verl GRPO/DAPO stack across four Qwen3 settings, evaluating AIME24/25/26 Avg@16.The 1.7B and 8B-Base runs use Tmax = 8k with n = 8 rollouts, while 30B-A3B-Base uses 16k with n = 16.
  • Main Results: CPPO achieves the best AIME24/25/26 Avg@16 across all four settings, scoring 31.88, 12.78, 31.11, and 54.79, respectively.Its margins over the second-best method are 3.06, 0.91, 1.39, and 5.56 absolute points.
  • Main Results: Under matched Top-K reduced-TV scores and threshold scales, CPPO improves on DPPO by 3.69, 1.88, 2.22, and 5.56 points across the four models.The comparison isolates the effect of weighted and prefix constraints rather than a different divergence measure or looser threshold scale.
  • Main Results: CPPO’s largest improvement is 5.56 points on Qwen3-30B-A3B-Base, where CISPO collapses during training and TRM-Max degrades to 20.27.This is the only 16k-rollout-horizon run, where early-token deviations propagate through a longer suffix.
  • Ablations: Ablations show CPPO remains above DPPO across nearby (δb, wmin) settings, with KL and Binary-TV variants achieving performance comparable to TV and consistently outperforming DPPO.The default sensitivity setting is (δb, wmin) = (0.02, 0.8); the KL control uses δ = 0.1 and δb = 0.002.

5 Conclusion · A Related work

CPPO replaces uniform token-level trust regions with position-aware masking and cumulative prefix budgets derived from a finite-horizon policy-improvement bound. Related work distinguishes methods by their policy-movement statistic and constraint placement, positioning CPPO as a prefix-budgeted method using an existing token-level divergence.

  • 5 Conclusion: CPPO addresses autoregressive asymmetry by tightening early-position constraints and tracking cumulative prefix deviations that can compound along the generated sequence.These mechanisms are derived from a prefix-constrained policy-improvement bound.
  • 5 Conclusion: Both CPPO mechanisms operate through masking alone, allowing reuse of the PPO/GRPO ratio-advantage objective and DPPO’s per-token divergence without additional loss terms.The method changes update eligibility rather than the underlying surrogate objective or divergence definition.
  • 5 Conclusion: Across four Qwen3 settings covering dense and MoE models plus Base and post-trained checkpoints, CPPO achieves the best validation AIME24/25/26 Avg@16.Ablations attribute the improvement to the position weight and prefix budget rather than the divergence estimator.
  • A Related work: CPPO is characterized as prefix-budgeted, position-aware token masking under a fixed token-level divergence statistic, rather than a new divergence measurement.Related work is organized by the statistic used to constrain policy movement.
  • A Related work: Sampled-ratio methods constrain updates with the sampled-token importance ratio, while GRPO adapts PPO to verifier-based group-relative advantages and later methods modify normalization, estimators, or clipping.The cited methods include PPO, GRPO, Dr.GRPO, REINFORCE++, and DAPO.
  • A Related work: Distributional-divergence methods use token-level TV or KL statistics, with DPPO applying uniform thresholds and TRM excluding whole responses when maximum or mean divergence exceeds a threshold.The matched comparison uses a Top-K reduced-TV statistic for DPPO and CPPO.
  • A Related work: CPPO is complementary to methods that reweight token updates, analyze entropy, or soften violations because it changes how deviation is allocated and accumulated across positions and prefixes.Its violation score also admits a soft variant, while prefix-ratio objectives such as MinPRO are not divergence-budget methods.

B Full Proofs and Theoretical Details · B.1 Finite-horizon performance difference identity · B.2 Suffix TV via maximal coupling

Appendix B supplies full proofs for the paper’s Section 2 results, including CPPO’s relationships to position-agnostic and sequence-level methods and a product-form suffix bound. It derives the finite-horizon performance difference identity and establishes the suffix total-variation bound through maximal coupling.

  • B Full Proofs and Theoretical Details: Appendix B provides full proofs of Section 2 results, related corollaries, and the product-form suffix bound.The corollaries characterize CPPO’s relationship to position-agnostic and sequence-level methods.
  • B.1 Finite-horizon performance difference identity: Lemma 2 states the exact finite-horizon performance difference identity from Section 2.1 under common support.The passage identifies this result as Equation (1) in Section 2.1.
  • B.1 Finite-horizon performance difference identity: The proof derives the identity by importance sampling and reverse telescoping.These proof steps connect the displayed terms to the performance-difference expression.
  • B.1 Finite-horizon performance difference identity: The resulting lower bound follows from replacing −∆(µ, π) with its absolute-value lower bound.The passage explicitly gives −∆(µ, π) ≥−|∆(µ, π)|.
  • B.2 Suffix TV via maximal coupling: Lemma 3 bounds suffix total variation when each later-step divergence satisfies Dj ≤ℓj pathwise for all j > t.The bound is stated conditionally for any st+1.
  • B.2 Suffix TV via maximal coupling: The suffix-TV proof constructs a stepwise maximal coupling that keeps both processes on the same state until their suffixes disagree.At each shared state, the one-step disagreement probability controls the coupling argument.
  • B.2 Suffix TV via maximal coupling: Using the coupling characterization of total variation and a union bound, the proof obtains the suffix bound with ℓj ∈[0, 1].The restriction ℓj ∈[0, 1] follows because total variation is at most one.

B.3 Remaining-horizon bound on the surrogate residual

Proposition 4 bounds the surrogate residual under pathwise token-level divergence thresholds using a maximal-coupling argument on the suffix likelihood ratio. The proof conditions on the next state, interprets the future likelihood ratio as a Radon–Nikodym derivative, and sums contributions through the final zero term.

  • Remaining-horizon bound: Proposition 4 establishes a surrogate-residual bound when D_t ≤ ℓ_t pathwise for every t.The result is identified as the first inequality of Equation (4) in Section 3.1.
  • Remaining-horizon bound: The proof uses maximal coupling on the suffix likelihood ratio to derive the bound.For fixed t < T and conditioned on s_{t+1}, the future ratio ρ_{t+1:T} is the Radon–Nikodym derivative of the suffix law under π relative to µ.
  • Remaining-horizon bound: The sampled-token ratio contributes to the bound, with the final term t = T equal to zero because ρ_{T+1:T} = 1.Summing over t = 1, . . . , T − 1 completes the claim.

B.4 CPPO policy-improvement bound (Theorem 1) · B.5 Linear schedule satisfies the monotonicity

Theorem 1 bounds the CPPO policy-improvement residual through weighted token-level divergences, expected prefix inequalities, and Abel summation. Proposition 5 shows that the implemented linear schedule makes the relevant threshold strictly decreasing, so Theorem 1 applies.

  • B.4 CPPO policy-improvement bound (Theorem 1): Theorem 1’s proof first reduces the residual to a weighted sum of token-level divergences under the constraint w_tD_t ≤ c_t.This is the first proof step.
  • B.4 CPPO policy-improvement bound (Theorem 1): The proof then converts prefix-budget constraints into prefix inequalities in expectation for every prefix length m = 1, . . . , T − 1.The conversion is performed by taking expectations of the prefix-budget constraints.
  • B.4 CPPO policy-improvement bound (Theorem 1): It defines a centered prefix slack before applying Abel summation to the resulting expressions.The centered prefix slack is introduced between the expectation step and Abel summation.
  • B.4 CPPO policy-improvement bound (Theorem 1): Because r_t is non-increasing, r_t − r_{t+1} ≥ 0, r_{T−1} ≥ 0, and S_t ≤ 0, the residual bound’s right-hand side is non-positive.The proof uses these sign conditions to establish the residual bound.
  • B.4 CPPO policy-improvement bound (Theorem 1): Combining the residual bound with Lemma 2 and substituting it into the resulting bound yields Equation (7).The proof explicitly performs these two final combination and substitution steps.
  • B.5 Linear schedule satisfies the monotonicity: Proposition 5 defines the linear schedule w_t = 1 − (1 − w_min)(t − 1)/(T − 1) and g_t = (T − t)/w_t.The proposition analyzes these schedule quantities to verify Theorem 1’s monotonicity requirement.
  • B.5 Linear schedule satisfies the monotonicity: g_t − g_{t+1} = w_min/(w_tw_{t+1}) > 0, so r_t = 4ξ̄ℓg_t is strictly decreasing for t = 1, . . . , T − 1.The same conclusion is stated directly as strict decrease of both g_t and r_t.
  • B.5 Linear schedule satisfies the monotonicity: Therefore, the implemented linear schedule satisfies the monotonicity assumption required for Theorem 1 to apply.This is the proposition’s stated consequence of the strict decrease of r_t.

B.6 Corollaries: uniform threshold and implementation threshold

Corollaries 6 and 7 compare residual divergence constants for uniform and position-dependent token-level thresholds. CPPO is tighter under δb < δ ≤ 1 for the uniform case and δb < δwmin for the position-dependent case, while the experimental linear schedule satisfies Theorem 1’s monotonicity assumption.

  • Uniform threshold: Corollary 6 identifies the uniform-threshold case as ct = wtδ, equivalently ℓt ≡ δ, with ¯ℓ = δ.
  • Uniform threshold: CPPO is tighter than the uniform-threshold comparison in the intended regime δb < δ ≤ 1.
  • Implementation threshold: Corollary 7 bounds the position-dependent case by ¯ℓ ≤ δ/wmin when wtDt ≤ δ, equivalently ℓt = δ/wt with wt ∈ [wmin, 1].
  • Implementation threshold: The implemented CPPO constant is tighter than Cuniform(δ) = 2ξT(T −1)δ2 whenever δb < δwmin.
  • Implementation threshold: The experimental linear schedule wt = 1 −1−wmin over T−1 positions satisfies Theorem 1’s monotonicity assumption by Proposition 5.

B.7 Implementation slack from the initial prefix

The implemented constraint introduces a constant initial prefix slack because the first ratio deviation matches the uniform token-level baseline threshold. For the linear schedule, this slack is bounded by δ and contributes an additional residual term O(T ℓ̄δ) without changing the leading prefix-budget term.

  • Initial prefix slack: The first sampled-token ratio update away from one matches the uniform token-level divergence baseline threshold, introducing constant initial slack that changes constants but not the leading prefix-budget term.This slack is added to the clean prefix inequality used in Theorem 1.
  • Initial prefix slack: Under the weighted constraint and monotonicity condition, the prefix terms satisfy w_j u_j ≤ δ_b W_m + η for m = 1, . . . , T − 1, with constant slack η ≥ 0.This is the initial prefix slack bound stated in Proposition 8.
  • Implemented schedule: For the implemented linear schedule, η ≤ δ, so the initial prefix slack contributes an additional residual term O(T ℓ̄δ).The schedule has w_1 = 1, and taking expectations yields the proposition’s prefix-slack condition η ≤ δ.

B.8 Sequence-level methods as special cases … C.3 Default gate and SAPO compatibility

The appendix relates CPPO to sequence-level and product-form trust-region bounds, explains Abel-summation tightness, and develops soft-gate interpretations, guarantees, and implementation choices. It also records the technical lemmas underlying these derivations and clarifies compatibility with SAPO-style attenuation.

  • B.8 Sequence-level methods as special cases: Removing position weights and the prefix budget reduces CPPO to the uniform token-level threshold, whose sequence-level form is TRM-Max.With w_t ≡ 1 and no prefix budget, only D_t ≤ δ remains; the implication is D_t ≤ δ ⇒ u_t ≤ δ for every t.
  • B.9 Product-form suffix bound: The product-form suffix branch yields a true O(T) bound when the relevant λ_tβ_t/w_t sequence is non-increasing.For small δ, β_t ≈ (T − t)δ, so this monotonicity coincides with the linear-branch condition; the paper does not use it as its main quantitative claim.
  • B.10 Abel summation and tightness: Abel summation converts the residual into prefix slacks weighted by coefficient gaps, tightening the bound when r_t = λ_t/w_t is non-increasing.The bound is tight for this prefix-constraint class because it is attained by u_t = δb whenever r_t is non-increasing.
  • B.11 Technical Lemmas: The technical lemmas provide total-variation and L1 identities, likelihood-ratio and coupling characterizations, weighted averaging, and Abel-summation tools.The weighted averaging equality condition is that a_t/w_t is constant, while maximal coupling attains the total-variation infimum.
  • C Soft-gate details: The soft-gate appendix extends the hard CPPO mask with gradient attenuation near constraint boundaries and discusses its empirical hard-versus-soft evaluation.The main paper focuses on the hard mask; the soft variant is evaluated in the Section 4.3 ablation.
  • C.1 Gradient-scaling interpretation: A normalized CPPO violation score enables smooth gradient scaling while preserving bounded effective normalized violation under admissible gates.The gate equals one when constraints hold and must satisfy g(x) = 1 for x ≤ 1 and xg(x) ≤ 1 for x > 1.
  • C.2 Mixture-policy construction: A mixture policy π_g = (1 − g_s)μ + g_sπ converts soft attenuation into an exact-TV guarantee with D_t replaced by g_tD_t.This construction is not deployed experimentally and distinguishes formal soft-gate guarantees from effective-gradient scaling.
  • C.3 Default gate and SAPO compatibility: The default inverse gate is g_inv(x) = min{1, 1/x}, while any gate satisfying xg(x) ≤ 1 preserves the formal guarantee and permits SAPO-style attenuation.SAPO-style schemes can apply their attenuation function to the normalized CPPO score instead of a sampled-ratio score.

D Experiment details and per-benchmark breakdown

The experiments use a GRPO/DAPO training stack with specified rollout, optimization, baseline, and evaluation protocols. Diagnostics show that CPPO’s prefix-budget masking is concentrated early in training while effective thresholds remain below token-level limits.

  • Training stack: All runs use a verl-compatible GRPO/DAPO trainer with group-normalized advantages, AdamW, maximum learning rate 1 × 10−6, and no entropy or KL-to-reference penalty.Prompts with zero advantage variance are skipped via the mask_std_0 filter.
  • Evaluation: Validation reports AIME24, AIME25, and AIME26 Avg@16 from 16 sampled completions, plus their unweighted mean AIME24/25/26 Avg@16.Reported summary scores are selected as the highest mean attained on the validation curve within [0, T stop].
  • Baselines and divergence scores: GRPO and CISPO use sampled importance ratios, whereas TRM-Max and TRM-Avg use token-level KL divergence with thresholds recommended by their original papers.The GRPO baseline uses Clip-Higher with asymmetric thresholds (ϵlow, ϵhigh) = (0.2, 0.28).
  • Base-model warm-up diagnostics: During Base-model warm-up, effective thresholds remain below token-level thresholds δ = 0.15 or 0.20, while prefix-budget masking is concentrated early in training.Average token-level divergence is initially exceptionally large and decays rapidly as the policy stabilizes.
  • Full training diagnostics: Figures 8–11 provide per-model traces for benchmark validation scores, reward, response length, and rollout-training relative log-probability error without changing checkpoint selection rules.The figures cover Qwen3-1.7B post-trained, Qwen3-1.7B-Base, Qwen3-8B-Base, and Qwen3-30B-A3B-Base.
Loading 2606.10968v2…