PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Adaptive Clipping Policy Optimization

Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft

arXiv:2609.00444v1cs.LGcs.CL

TL;DR

Fixed clipping in RLVR suppresses scarce, high-advantage correct rollouts even though they provide stronger learning signals. GAPO adapts only the clipping boundary to rollout advantage using a reverse-KL trust-region motivation, while preserving the standard PPO/GSPO surrogate and requiring no reward shaping. Across several models and math and coding benchmarks, it consistently improves pass@1 and pass@k while retaining strong IS–advantage correlation.

  • Problem

    Fixed clipping suppresses scarce, high-advantage correct rollouts at rates comparable to abundant correct rollouts despite their stronger learning signal.

  • Method

    GAPO is a plug-in group-relative optimization method that adapts each rollout’s clipping threshold to advantage through the group correctness count while preserving the standard PPO/GSPO surrogate.

  • Results

    GAPO consistently improves pass@1 and pass@k across Qwen, Llama, and DeepSeek models on mathematical reasoning and competitive coding benchmarks.

  • Takeaways & Limitations

    Adaptive clipping preserves stronger update headroom for higher-advantage rollouts without reward shaping and maintains high IS–advantage correlation throughout training.

  • Takeaways & Limitations

    The practical clipping schedule treats the reverse-KL multiplier as global, although the underlying batch-level constraint is shared across multiple prompts or tasks.

Abstract

from arXiv · show

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.

1 Introduction

RLVR’s fixed clipping boundary suppresses scarce, high-advantage correct rollouts as aggressively as abundant, lower-signal rollouts. GAPO addresses this mismatch by adapting clip thresholds to advantage using a reverse-KL trust-region rationale, while preserving the standard surrogate and improving pass@1 and pass@k.

  • Problem: Scarcely correct rollouts exhibit higher IS ratios and reach a fixed clipping boundary earlier than rollouts from easier problems.Their IS ratios also continue growing at a rate equal to or exceeding that of abundant correct rollouts.
  • Problem: Fixed IS-ratio clipping treats rollouts with different advantages uniformly, suppressing scarce correct rollouts despite their stronger learning signal.Low-correctness groups correspond to harder problems and higher-advantage correct rollouts.
  • Problem: Uniformly widening the clipping boundary, as in DAPO, does not resolve the core problem because clip width remains uniform across different advantage levels.The proposed change concerns relative clip width across levels of group success, not merely absolute boundary width.
  • Motivation: A reverse-KL trust-region perspective motivates giving rollouts with larger learning signals proportionally greater update headroom.The trust-region-optimal IS ratio scales exponentially with advantage and yields a closed-form RLVR clipping rule indexed by group correctness count.
  • Method: GAPO is a plug-in PPO/GSPO modification that adapts only the clipping boundary while preserving the standard surrogate objective and direct pass@1 optimization.It uses the group statistic c to set per-rollout thresholds and requires no reward shaping.
  • Results: GAPO consistently improves pass@1 and pass@k across Qwen, Llama, and DeepSeek models on mathematical reasoning and code-generation benchmarks.The method also maintains strong correlation between IS ratio and advantage throughout training.

2 Preliminaries

RLVR trains language models with binary verifier rewards, and group-relative methods assign advantages according to how many rollouts in a group are correct. PPO-style clipping then constrains policy updates, but fixed boundaries disproportionately suppress high-advantage rollouts.

  • RL with Verifiable Rewards: RLVR samples a complete response for each prompt and assigns it a binary verifiable reward from an automated verifier.The training objective is the expected reward.
  • Group-Relative Advantages: Group-relative methods assign correct-rollout advantages from the group’s correctness count, producing larger advantages for scarce successes and smaller advantages for abundant successes.For c=1, the advantage is (k−1)/k; for c=k−1, it is 1/k.
  • Clipping: Standard PPO-style clipping applies the same trust-region interval [1−ϵ, 1+ϵ] to every rollout.This uniform boundary suppresses high-advantage rollouts because their importance ratios cross the boundary fastest.
  • Trust Regions: The trust-region analysis permits a reverse-KL formulation because total-variation distance is symmetric and Pinsker’s inequality bounds it from either direction.The empirical batch expectation replaces the worst-case maximum for tractability, as in PPO.

3 Group Adaptive Policy Optimization

The method derives per-prompt trust-region updates and converts the resulting advantage-dependent IS optimum into an adaptive clipping rule. GAPO gives scarce correct rollouts more update headroom while preserving GSPO’s sequence-level objective and ratio.

  • 3.1 Per-Prompt Trust Region Optimization: Per-prompt optimization yields an optimal IS ratio that determines how far each rollout’s probability should be increased under the trust-region formulation.Pushing beyond this ratio either violates the trust region or misallocates probability relative to alternatives.
  • 3.1 Per-Prompt Trust Region Optimization: The practical derivation uses a large λ approximation, making the adaptive threshold nearly linear in rollout advantage while remaining conservative at token-IS scale.The token-level maximum upper clip is 0.28, with relative error on Ai/λ bounded between 2% and 13%.
  • 3.2 GAPO Adaptive Clip Formula: GAPO’s upper clip increases as group correctness decreases: scarce correct rollouts receive maximum headroom, while abundant correct rollouts receive the minimum.Incorrect rollouts always use ϵlo because the upper clip is irrelevant for negative advantage.
  • 3.2 GAPO Adaptive Clip Formula: For negative-advantage rollouts, adaptive lower clipping primarily protects the trust region for penalizing incorrect rollouts on easy problems and is therefore fixed.It does not alter the positive signal or the clipping bias affecting exploration.
  • 3.2 GAPO Adaptive Clip Formula: GAPO combines its adaptive clip with GSPO’s sequence-level importance ratio, whose threshold directly gates the rollout-level ratio derived from the same trust-region argument.The sequence-level ratio is constructed from per-token ratios using their geometric mean.

4 Experiments & Results

Experiments evaluate GAPO’s training dynamics, benchmark performance, and design choices across multiple models and mathematical, coding, and instruction-following tasks. GAPO preserves IS–advantage alignment and generally improves pass@1 and pass@k, particularly on harder benchmarks.

  • Experiment Setup: GAPO is evaluated on three base models spanning two model families and two pretraining regimes, using mathematical, coding, and instruction-following benchmarks.The setup includes Qwen2.5-Math-1.5B, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B.
  • Experiment Setup: GAPO preserves the standard surrogate while adapting clipping to rollout advantage and is compared with fixed-clip and advantage-shaping baselines at matched clip widths.The comparisons include GRPO, Dr.GRPO, symmetric and asymmetric GSPO, F-GRPO, and F-GSPO.
  • Adaptive clipping preserves the IS–advantage correlation: GAPO maintains high IS–advantage correlation after clipping activates, whereas fixed clipping collapses the correlation; late-training performance also remains higher on AIME24.GAPO maintains r > 0.8 in late training, while fixed-clip baselines degrade toward negative correlation.
  • Benchmark Results: Across three base models, GAPO improves pass@1 without sacrificing pass@k, with gains concentrated on harder benchmarks.On AIME24, the reported improvement over asymmetric GSPO is +1.72 with p<0.001.
  • Ablation Study: Sequence-level adaptive clipping outperforms token-level adaptive clipping on AIME24 and math-average pass@1, while relaxing ϵmax hi from 5e−3 to 5.25e−3 yields a small improvement.Sequence-level GAPO reports AIME24 pass@1 17.9% and math average pass@1 37.9%.

5 Related Work

Related work covers fixed-clip and alternative group-relative RLVR methods, direct pass@k optimization, and advantage-shaping approaches. GAPO differs by deriving advantage-proportional clipping while retaining direct pass@1 optimization.

  • RLVR: Dr.GRPO, DAPO, and GSPO modify normalization, clipping asymmetry, or IS granularity, while GAPO derives a clip threshold proportional to advantage from a reverse-KL perspective.The related methods retain fixed clipping or alter aggregation, whereas GAPO changes the relative clip width across rollouts.
  • Pass@k optimization: Methods that directly optimize pass@k address large-k performance, but prior work reports that pass@1 can degrade under this training direction.GAPO instead retains direct pass@1 optimization while encouraging exploration through IS–advantage alignment.
  • Advantage shaping: F-GRPO and F-GSPO use advantage shaping based on within-group success count, providing an alternative to GAPO’s adaptive clipping boundary.The experiments compare these focal-shaping variants with fixed-clip RLVR methods at matched clip widths.

6 Conclusion

GAPO adapts per-rollout clipping using a reverse-KL trust-region motivation while preserving the standard surrogate and directly optimizing pass@1. Across evaluated models and benchmarks, it improves pass@1, retains pass@k, and maintains IS–advantage correlation.

  • GAPO is a plug-in method that adapts the per-rollout clipping threshold to advantage without introducing new hyperparameters beyond the existing clip range.In RLVR, binary rewards reduce the schedule to a closed form indexed by the group correctness count c.
  • The reverse-KL trust-region analysis motivates clipping thresholds that scale with rollout advantage rather than applying one boundary uniformly.The proposed schedule is derived from a trust-region-optimal importance-sampling ratio at a single prompt.
  • GAPO consistently improves pass@1 while retaining pass@k across mathematical reasoning and competitive coding benchmarks.The evaluations include Qwen2.5-Math-1.5B, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B.
  • GAPO maintains high IS–advantage correlation throughout training where fixed clipping collapses it.The method implicitly encourages exploration while continuing to optimize pass@1 directly.

Limitations

The limitations concern GAPO’s fixed treatment of negative-advantage rollouts and its single-prompt trust-region derivation. The latter requires a global multiplier approximation in multi-prompt batches, although empirical correlations suggest the heuristic remains useful.

  • GAPO adapts only the upper clipping threshold, leaving the lower threshold for incorrect rollouts fixed across rollouts.The authors leave joint adaptation of negative-rollout headroom for future work because it may affect exploration behavior indirectly.
  • The per-rollout optimal importance-sampling ratio is derived under a reverse-KL trust region at a single prompt.This derivation is an approximation for the multi-prompt batches used in RLVR training.
  • In batch-level KL constraints, the multiplier is shared across prompts in principle, but GAPO treats it as a global constant so the rule depends only on c.The authors note that the gap is real while observed IS–advantage correlations indicate useful empirical guidance.

Ethical Considerations

GAPO is presented as a methodological RLVR contribution using standard data, models, benchmarks, and training protocols. The authors identify only general dual-use considerations associated with improving language-model reasoning capabilities.

  • GAPO adapts clipping thresholds without adding model capabilities, training data, or evaluation protocols beyond standard RLVR practice.Its intended impact is improved training efficiency and final performance on mathematical reasoning, including educational applications.
  • The experiments use publicly available training data, evaluation benchmarks, and openly released base models.No human subjects, annotators, or proprietary data were involved.
  • GAPO may indirectly contribute to more capable reasoning systems, carrying general dual-use considerations common to language-modeling research.The authors do not anticipate risks beyond those inherent to RLVR training in general.
  • A checkpoint intervention branches from GSPO-asym at step 600 to isolate uniform clipping as the cause of the observed correlation decline.The adaptive branch sustains advantage–IS correlation and Pass@256 while the uniform branch declines.

C Robustness Analysis

Robustness analyses examine training-seed variability and clip-width sensitivity. GAPO’s Pass@1 gains are statistically significant on at least three of six benchmarks against every baseline, and its adaptive clipping outperforms uniform clipping across tested levels.

  • The robustness analysis tests statistical significance across training seeds and sensitivity to the clip-width hyperparameter.Three-seed runs cover the Qwen2.5-1.5B-Math setting because compute constraints prevented three seeds for all configurations.
  • GAPO’s Pass@1 gains are significant on at least 3 of 6 benchmarks against every baseline across 3-seed comparisons.The strongest gains occur on harder reasoning benchmarks such as AIME24.
  • GAPO’s adaptive upper clipping outperforms uniform clipping at every tested clip-width level on AIME24 Pass@1.The comparison uses Qwen2.5-1.5B-Math with the lower threshold fixed at 3e−3.

D Per-Problem Solve Rate Analysis

On AIME24 validation, GAPO’s per-problem solve-rate advantage over the baseline is concentrated on medium-difficulty problems and increases late in training.

  • GAPO’s solve-rate advantage is concentrated on medium-difficulty problems, specifically rows 5–12.Problems are sorted from easiest at the top, and blue denotes higher solve rate for GAPO.
  • The advantage grows in late training as the baselines progressively lose ground.The figure tracks Qwen2.5-Math-1.5B over training using 256 samples per problem and evaluations every 10 steps.
  • The plotted differences are averaged across all seeds for each method.

E Diverse Correct Solutions Found by GAPO

GAPO preserves more diverse correct solutions than the baseline on an AIME24 problem, while the analyzed generations independently derive the same answer, 902.

  • Generation 1: Generation 1 counts 900 mixed row-and-column configurations plus 2 monochrome cases, yielding answer 902.It assigns colors to rows and columns, placing chips where their colors agree.
  • Generation 2: Generation 2 reaches the same answer, 902, by partitioning the grid into color blocks and counting 900 non-trivial splits plus 2 trivial cases.The non-trivial count is 30 × 30, with the two monochrome configurations added separately.
  • Diverse correct solutions: GAPO produces 28 correct generations on one AIME24 problem, compared with 3 for baseline GSPO.Near peak validation pass@1, GAPO samples three distinct correct reasoning paths.
  • Diverse correct solutions: The three GAPO reasoning paths converge to the same answer while approaching the problem from different conceptual angles.This contrasts with the baseline’s effective collapse of correct generations into fewer modes.
Loading 2609.00444v1…