PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Adaptive Clipping Policy Optimization

Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, Rein Houthooft

arXiv:2609.00444v1cs.LGcs.CL

TL;DR

Fixed clipping applies a uniform boundary to rollouts with different learning signals; GAPO adapts the clipping threshold to rollout advantage. Across tested models and math and code-generation benchmarks, it consistently improves Pass@1 and Pass@k.

  • Problem

    Fixed clipping applies a uniform boundary even though rollouts with different advantages carry substantially different learning signals.

    Source: §1 · pp. 1–2

  • Method

    GAPO adapts per-rollout clipping to advantage, using a reverse-KL trust-region rationale and an RLVR schedule indexed by group correctness count c.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    GAPO consistently improves Pass@1 and Pass@k across Qwen2.5-Math-1.5B, Llama-3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B on math and code-generation benchmarks.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Across training, GAPO maintains high IS–advantage correlation, whereas fixed clipping collapses it.

    Source: §6 · p. 9

    Its theoretical motivation is per-prompt, while the batch-averaged KL constraint implies a multiplier shared across prompts; using a global constant leaves a real gap.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.

1 Introduction

GAPO adapts clipping to rollout advantage because fixed boundaries suppress scarce, high-signal correct rollouts disproportionately. It preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold, and the paper reports consistent pass@1 and pass@k improvements across its evaluated models and benchmarks.

1 Introduction

  • With fixed clip boundaries, scarce correct rollouts carry stronger learning signals yet are clipped at rates comparable to abundant correct rollouts.Their IS ratios are higher and continue growing at least as fast, so they hit the clip first and are disproportionately suppressed early.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Uniformly widening the clip boundary, as in DAPO (Yu et al., 2025), does not address the problem because the boundary remains uniform across different advantage levels.Group-relative methods assign correct rollouts advantage (k-c)/k, distinguishing scarce high-advantage from abundant low-advantage successes.Source: §1 · pp. 1–2
  • Reverse-KL trust-region reasoning motivates giving rollouts with larger advantages proportionally greater update headroom.The proposed trust-region-optimal IS ratio scales exponentially with advantage, and in RLVR the clipping rule can be computed from group success count c.Source: §1 · p. 2
  • GAPO adapts per-rollout clipping thresholds to advantage using group success, while preserving the standard PPO/GSPO surrogate and directly optimizing pass@1.It modifies only the clipping boundary and requires no reward shaping.Source: §1 · p. 2
  • GAPO consistently improves pass@1 and pass@k across three models on mathematical reasoning and code generation benchmarks.The paper also reports a strong correlation between IS ratio and advantage throughout training and a checkpoint intervention to address common confounds.Source: §1 · p. 2

2 Preliminaries

The preliminaries define RLVR and group-relative advantages, then motivate adaptive clipping by showing why fixed bounds suppress high-advantage rollouts. They also establish reverse-KL trust regions as a basis for the policy-update guarantee.

2.1 RL with Verifiable Rewards

  • In RLVR, a language model response to a prompt receives a binary verifiable reward from an automated verifier, and training maximizes expected reward.Source: §2.1 · p. 2
  • Scarce correct rollouts (c=1) receive advantage (k−1)/k, versus 1/k for abundant correct rollouts (c=k−1), while fixed clipping disproportionately suppresses high-advantage rollouts.The figure contrasts uniform symmetric and asymmetric clipping with GAPO’s per-c adaptive boundary, which keeps updates proportional to advantage as clipping begins.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 Trust Region Policy Optimization

  • The policy-improvement guarantee remains valid when the forward KL term is replaced by reverse KL, because total-variation distance is symmetric and Pinsker’s inequality bounds it from either side.Maximizing the bound’s right-hand side guarantees monotonic improvement of J(θ) at each update.Source: §2.2 · p. 3
  • For tractability, the constrained optimization uses the empirical batch expectation in place of the worst-case maximum, as in PPO (Schulman et al., 2017).Source: §2.2 · p. 3

3 Group Adaptive Policy Optimization

GAPO derives a per-prompt trust-region clipping rule that gives rollouts with stronger learning signals more update headroom, then applies an adaptive upper clip to GSPO’s rollout-level sequence ratio.

3.1 Per-Prompt Trust Region Optimization

  • The per-prompt trust-region derivation identifies an optimal importance-sampling ratio; pushing a rollout beyond it violates the constraint or leaves competing responses under-updated.The optimization is formulated for a prompt x, which makes the batch-level constraint per-prompt and exact only for single-prompt batches.Source: §3.1 · pp. 3–4
  • With GSPO clip widths ϵ∼10−3, GAPO sets ϵhi∼10−3; at token-IS scale, ϵmaxhi=0.28 is conservative and the linearization’s relative error is 2–13%.The linearization is for presentation cleanliness and is essentially exact at the sequence-IS level.Source: §3.1 · p. 4

3.2 GAPO Adaptive Clip Formula

  • Scarce correct rollouts (c=1) receive maximum upper-clip headroom, abundant correct rollouts (c→k) receive the minimum, and incorrect rollouts use fixed ϵlo.The paper fixes ϵlo because it does not affect the positive signal or the clipping bias that prevents exploration (Yu et al., 2025).Source: §3.2 · p. 4
  • GAPO combines its adaptive clip with GSPO’s sequence-level importance ratio, the geometric mean of per-token ratios, and applies the threshold at the rollout level.The clip gates the sequence-level ratio derived from the same per-rollout trust-region argument.Source: §3.2 · pp. 4–5

4 Experiments & Results

GAPO is evaluated across model families, training regimes, math and coding tasks, with benchmark results and training dynamics showing how adaptive clipping affects performance and the IS–advantage relationship. The experiments also test potential confounds and key design choices.

4.1 Experiment Setup

  • Experiments span Qwen, Llama, and reasoning-distilled DeepSeek models, with RLVR on filtered math and coding datasets and fixed-clip, asymmetric-clipping, and advantage-shaping baselines.Evaluation includes math and coding benchmarks plus IFEval as an out-of-domain instruction-following probe; comparisons include matched clip widths, and GSPO is the strongest non-adaptive baseline in this setting.Source: §4.1 · p. 5

4.2 Adaptive clipping preserves the IS–advantage correlation

  • GAPO preserves IS–advantage correlation as clipping rises, while fixed clipping suppresses scarce correct rollouts at the same rate as abundant ones.Late in training, GAPO maintains r > 0.8; fixed-clip correlations degrade toward negative values. GAPO also sustains higher pass@1 and pass@256, and retains 10.9 AIME24 problems with > 5% solve rate versus 6.7–8.6 for baselines.Source: Figure 5 · p. 6, §4.2 · p. 6, Figure 4 · p. 6

4.2.1 Ruling out confounds before clipping

  • Switching the step-600 checkpoint to adaptive clipping preserves higher advantage–IS correlation and pass@k, ruling out pre-clipping factors as the cause of the decline.The intervention followed aggressive clipping by the GSPO-asymmetric uniform-clipping baseline.Source: §4.2.1 · p. 6

4.3 Benchmark results

  • 37.9/76.3 average pass@1/pass@256 for Qwen and 24.1/63.4 for Llama; GAPO also leads or matches pass@1 on all six DeepSeek math benchmarks.Qwen’s highest AIME24 pass@1 is 17.9, versus 15.4 for F-GSPO and 16.2 for asymmetric GSPO; Llama’s is 14.6, versus 10.7–14.4 for fixed-clip GSPO variants. On DeepSeek, AIME24 is 44.0/76.7 (+2.7/+3.4 over GSPO) and AIME25 is 30.8/56.7 (+1.4/+6.7 over GSPO). Across the three models, gains concentrate on harder benchmarks; AIME24 improves by +1.72 over asymmetric GSPO (p<0.001), while AMC regresses against that baseline.Source: Table 1 · p. 7, §4.3 · p. 7
  • GAPO remains effective on coding tasks, evaluated using the DeepCoder recipe with token-level importance sampling.The coding evaluation uses 24,269 DeepCoder samples, and the table compares DeepCoder-1.5B with GAPO-DeepCoder-1.5B.Source: §4.1 · p. 5, §4.3 · p. 7, Table 2 · p. 7

4.4 Ablation Study

  • Sequence-level GAPO exceeds token-level GAPO on AIME24 pass@1 (17.9% vs. 14.6%) and math average pass@1 (37.9% vs. 37.6%).Increasing group size k from 4 to 8 widens GAPO’s performance gap over GSPO; above 8, math training shows no noticeable gain but is more stable. Fixed upper clipping is worse than GAPO on AIME24, while relaxing ϵmax hi from 5e−3 to 5.25e−3 yields a small improvement.Source: §4.4 · p. 7

5 Related Work

Related work spans LLM post-training and RLVR methods that preserve exploration through changes to clipping, rollout allocation, or advantage/reward shaping. GAPO instead derives advantage-proportional clipping from a reverse-KL trust-region perspective while retaining the pass@1 objective.

5.1 LLM Post-training

  • Prior LLM post-training spans SFT on high-quality reasoning traces, multi-teacher reasoning distillation, and token-level student–teacher distribution matching.The passage describes s1 (Muennighoff et al., 2025), SSFT (Jia et al., 2026), and on-policy distillation (Agarwal et al., 2024).Source: §5.1 · p. 7
  • Scaling training rollouts per group from k=4 to k=8 improves pass@1 and pass@256 on AIME24 evaluation and widens the gap between GSPO and GAPO.Figure 6 reports this ablation on the number of rollouts per group used during training.Source: Figure 6 · p. 8
  • GAPO derives an advantage-proportional clip threshold from the per-prompt reverse-KL trust-region-optimal IS ratio, unlike prior exploration methods using fixed clip thresholds or rollout/token changes.DAPO’s clip-higher and CISPO’s gradient preservation share the goal of retaining exploration tokens; other approaches scale rollouts, separate correct/incorrect clipping thresholds, or optimize exploratory forking tokens.Source: §5.1 · p. 8

5.2 Advantage and Reward Shaping for Exploration in RLVR

  • Prior work directly optimizing pass@k was motivated by RLVR-trained models scoring below the base model at large k, but pass@1 has been observed to degrade under that training.The cited works are Chen et al. (2025b), Walder and Karkhanis (2025), Thrampoulidis et al. (2025), and Barakat et al. (2026).Source: §5.2 · p. 8
  • GAPO retains pass@1 optimization while preserving gradients on rare correct rollouts through high correlation between empirical IS ratio and advantage, unlike dedicated advantage-shaping schemes.Prior schemes shape advantages using within-group success counts, reweight groups by success rate, or add a correctness-based differential-smoothing reward.Source: §5.2 · p. 9

6 Conclusion

GAPO is a plug-in adaptive clipping method that adjusts per-rollout thresholds according to advantage rather than applying a uniform clip. Across three models and math and coding benchmarks, it improves pass@1 while retaining pass@k and maintains IS–advantage correlation where fixed clipping collapses it.

6 Conclusion

  • Across three models on mathematical reasoning and competitive coding benchmarks, GAPO consistently improves pass@1 while retaining pass@k and maintains high IS–advantage correlation throughout training.Fixed clipping’s IS–advantage correlation collapses during training.Source: §6 · p. 9
  • GAPO uses a reverse-KL trust-region perspective in which the optimal importance-sampling ratio scales exponentially with advantage, motivating advantage-scaled per-rollout clip thresholds.With binary RLVR rewards, this yields a closed-form schedule indexed by group correctness count c and requires no hyperparameters beyond the existing clip range.Source: §6 · p. 9
  • Unlike reward- and advantage-shaping approaches, GAPO directly optimizes pass@1 while implicitly encouraging exploration through trust-region constraint allocation.The method is presented as a simple plug-in adaptive clipping modification.Source: §6 · p. 9

Limitations

GAPO leaves the lower clipping threshold for incorrect rollouts fixed and derives its per-rollout ratio from a single-prompt trust-region analysis, although the KL constraint is averaged across prompts.

Fixed ϵlo across rollouts

  • GAPO adapts only ϵhi for positive-advantage rollouts, while holding ϵlo fixed for incorrect rollouts.Future work will study jointly adapting the headroom for penalizing incorrect rollouts based on task difficulty and its effect on exploration.Source: Limitations · p. 9

Single prompt trust-region analysis

  • The per-rollout IS-ratio derivation assumes a single prompt, although the batch-averaged KL constraint implies λ is shared across prompts.The observed IS-advantage correlations in Figure 5 suggest that the per-prompt heuristic still provides useful empirical guidance.Source: Limitations · p. 9

Ethical Considerations

GAPO changes only the per-rollout clipping threshold and uses publicly available data, benchmarks, and openly released models, with no human subjects or proprietary data. Its broader impact is intended to improve LLM training efficiency and mathematical-reasoning performance, while carrying general RLVR dual-use considerations.

Scope and broader impact

  • GAPO adapts per-rollout clipping without adding model capabilities, training data, or evaluation protocols beyond standard RLVR, aiming to improve LLM efficiency and mathematical-reasoning performance.The intended applications include education.Source: Ethical Considerations · p. 9

Data and models

  • All listed training data, benchmarks, and base models are publicly available or openly released, and the work involved no human subjects, annotators, or proprietary data.Source: Ethical Considerations · p. 9

Dual-use considerations

  • GAPO may indirectly contribute to more capable reasoning systems, but it does not specifically lower harmful-content generation costs or target harmful-application capabilities.The authors anticipate no risks beyond those inherent to RLVR training in general.Source: Ethical Considerations · p. 9

A Training Hyperparameters · B Intervention study

An intervention from GSPO-asym step 600 attributes the decline in advantage–IS correlation and Pass@256 under uniform clipping to the clipping policy: GAPO sustains both, while uniform clipping declines.

B Intervention study

  • From GSPO-asym step 600, GAPO sustained advantage–IS correlation and Pass@256, while uniform clipping declined.The intervention rules out earlier factors, such as a reduced advantage spectrum, as the cause of the decline.Source: Figure 7 · p. 12, B Intervention study · p. 12

A Training Hyperparameters

  • Math RL used bfloat16, AdamW at 1 × 10−6, and 8xH200 GPUs with DP=8, TP=1, SP=1, and FSDP.Training took about 4–5 days for DS-R1 distilled models and 1 day for Qwen2.5-Math-1.5B and Llama models; Table 5 covers coding RL hyperparameters for Table 2.Source: B Intervention study · p. 12, Table 4 · p. 12, Table 5 · p. 12

C Robustness Analysis · C.1 Significance Tests · C.2 Hyperparameter Sweeps

The robustness analysis tests GAPO’s results across training seeds and clip-width settings. It reports significant Pass@1 gains against every baseline on at least 3 of 6 benchmarks and improvement over uniform clipping at every tested level.

C Robustness Analysis

  • The robustness analysis checks statistical significance across training seeds and sensitivity to the clip-width hyperparameter.Seeded runs use the Qwen2.5-1.5B-Math setting because compute constraints prevented three-seed runs for every configuration.Source: C Robustness Analysis · p. 13

C.1 Significance Tests

  • Table 6 isolates GAPO’s adaptive upper clipping threshold by matching each baseline’s importance-sampling scheme and clip range.GAPO adapts ϵhi per group while the baseline holds it fixed.Source: Table 6 · p. 13
  • Table 6 reports 3-seed means with 95% confidence intervals, while Table 7 reports paired per-seed differences.The seeded Table 6 means can differ slightly from the single-seed Table 3 entries.Source: Table 6 · p. 13, Table 7 · p. 13, C Robustness Analysis · p. 13
  • GAPO’s Pass@1 gains are significant on at least 3 of 6 benchmarks against every baseline, with the strongest improvement on harder benchmarks such as AIME24.The pairwise tests show one meaningful regression: against GSPO 3/5 on AMC.Source: Table 7 · p. 13, §C.1 · p. 13
  • The only meaningful regression reported is GAPO versus GSPO 3/5 on AMC.Other baseline comparisons show statistically significant gains on at least 3 of 6 benchmarks.Source: Table 7 · p. 13
  • Table 7 tests paired per-seed Pass@1 differences with Welch’s t-test, marking significance at p < 0.05, p < 0.01, and p < 0.001.Unmarked comparisons are not significant.Source: Table 7 · p. 13

C.2 Hyperparameter Sweeps

  • GAPO’s adaptive ϵhi beats uniform clipping at every tested level on AIME24 Pass@1 for Qwen2.5-1.5B-Math.The sweep fixes ϵlo at 3e−3 and pairs each uniform fixed-clipping GSPO baseline with a GAPO run.Source: §C.2 · p. 14, Table 8 · p. 14

D Per-Problem Solve Rate Analysis · E Diverse Correct Solutions Found by GAPO

GAPO sustains higher solve rates on medium-difficulty AIME24 problems late in training and preserves more diverse correct solutions than GSPO on an example problem.

D Per-Problem Solve Rate Analysis

  • Averaged over steps 1200–1530, GAPO maintains 10.9 problems with > 5% solve rate versus 8.6 (F-GSPO), 8.5 (GSPO asym), and 6.7 (GSPO sym).The analysis uses AIME24 validation for Qwen2.5-Math-1.5B, with 256 samples per problem and results averaged across seeds.Source: Figure 8 · p. 14
  • GAPO’s solve-rate advantage is concentrated on medium-difficulty problems (rows 5–12) and grows late as baselines lose solve rate on these frontier problems.Source: Figure 8 · p. 14
  • Figure 8 compares per-problem solve-rate differences between GAPO and baseline methods over training, with problems sorted from easiest at the top.Evaluation was conducted every 10 steps.Source: Figure 8 · p. 14

E Diverse Correct Solutions Found by GAPO

  • On one AIME24 problem, GAPO produces 28 correct generations, compared with 3 from baseline GSPO.Source: Figure 9 · p. 15
  • Near peak validation pass@1, GAPO samples three distinct correct reasoning paths on that problem, all converging to the same answer from different conceptual angles.Source: Figure 9 · p. 15
  • The example suggests GAPO retains multiple correct modes where baseline GSPO has effectively collapsed.Source: Figure 9 · p. 15

Related papers

Loading 2609.00444v1…