Clipping-Free Policy Optimization for Large Language Models

Ömer Veysel Çağatan; Barış Akgün; Gözde Gül Şahin; Xuandong Zhao

Source-linked AI summary

Clipping-Free Policy Optimization for Large Language Models

Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao

arXiv:2601.22801v1Jan 30, 2026cs.LG

TL;DR

Clipping-based RL objectives for LLM post-training can create zero-gradient regions and associated instability, reward exploitation, and capability degradation. CFPO replaces clipping with a TV-derived convex quadratic penalty and, across reasoning and alignment settings, extends stable training while maintaining competitive performance and reducing alignment failures. The method is presented as a drop-in alternative requiring only one code change and no additional hyperparameters.

Problem
Hard clipping in dominant LLM post-training algorithms creates zero-gradient regions associated with reward exploitation, capability degradation, entropy collapse, and training instability.
Method
CFPO replaces clipped objectives with an everywhere-differentiable convex quadratic penalty derived from Total Variation divergence constraints.
Results
Across reasoning and alignment, CFPO extends stable training, matches reasoning accuracy, improves length-controlled AlpacaEval by 4 points, and reduces alignment tax from 12–16% to 4–5%.
Takeaways & Limitations
CFPO is a promising drop-in alternative to clipping-based methods that maintains downstream performance while requiring one code change and no additional hyperparameters.
Takeaways & Limitations
Evaluation is limited to Qwen and LLaMA families, MATH and OpenRLHF datasets, and model sizes up to 8B; larger scales, architectures, domains, and sparse or noisy rewards remain untested.

Abstract

from arXiv · show

Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.

1. Introduction

Clipping-based objectives create zero-gradient regions and related optimization failures, motivating CFPO’s differentiable quadratic alternative. Across reasoning and alignment, CFPO extends stable training, preserves competitive performance, and reduces verbosity exploitation and alignment tax.

Motivation: Hard clipping creates zero-gradient regions that are linked to verbosity exploitation, capability degradation, entropy collapse, and training instability across scales and configurations.These failures are presented as intrinsic to clipping rather than incidental to particular implementations.
Method: CFPO eliminates clipping by replacing it with an everywhere-differentiable convex quadratic penalty derived from Total Variation divergence constraints.The penalty applies a restoring force rather than zeroing gradients beyond a threshold.
Reasoning results: CFPO matches GRPO on MATH500, AIME24, GSM8K, and GPQA-Diamond while extending the stable reasoning-training regime beyond GRPO’s collapse around 8 iterations and by 16 iterations.CFPO maintains controlled entropy decay, policy KL, and clipping ratio throughout training.
Alignment results: CFPO improves length-controlled AlpacaEval by 4 points, remains competitive on Arena-Hard and MT-Bench, and reduces alignment tax from 12–16% to 4–5%.The alignment results are reported alongside mitigation of verbosity exploitation.
Practical implication: CFPO is presented as a promising drop-in alternative that improves training stability without sacrificing downstream performance, requiring one code-line change and no additional hyperparameters.The paper describes this substitution as requiring no additional hyperparameters.

2. Background

PPO constrains policy updates with a clipped surrogate based on probability ratios, while GRPO adapts this mechanism without requiring a learned critic. SPO motivates replacing clipping with a TV-derived quadratic penalty that remains differentiable and retains nonzero restoring gradients.

Proximal Policy Optimization: PPO stabilizes policy-gradient training by maximizing a clipped surrogate objective that constrains single-update policy changes.The clipping range is controlled by the probability-ratio constraint.
Proximal Policy Optimization: The probability ratio r_t(θ) compares current and old policy probabilities, while estimated advantage Â_t weights the surrogate objective and ϵ controls the clipping range.Clipping removes incentives to push the ratio outside [1−ϵ, 1+ϵ].
Group Relative Policy Optimization: GRPO removes PPO’s comparable-size critic by estimating advantages from relative rewards among groups of sampled responses to the same prompt.For each prompt, GRPO samples multiple responses and compares their rewards within the group.
Simple Policy Optimization: PPO clipping can be interpreted as an approximate Total Variation trust-region constraint, while TV-constrained optimization provides a larger feasible policy space and a tighter improvement lower bound than the corresponding KL formulation.The stated comparison assumes δTV ≥ δKL/2 for the policy-improvement result.
Simple Policy Optimization: SPO replaces clipping with a convex quadratic penalty whose maximizer retains nonzero gradients and whose objective is differentiable in the probability ratio.Unlike clipping, the constraint is incorporated directly through the penalty.

3. Methodology

CFPO adapts SPO to LLM post-training by replacing clipped surrogate objectives with a convex quadratic penalty, while retaining critic-free optimization and modular advantage estimation. The method is evaluated with group-relative and leave-one-out advantages, with stability benefits reported across both settings.

3.1. Clipping-Free Policy Optimization: CFPO adapts SPO to language-model post-training by replacing the clipped surrogate objective with SPO’s quadratic penalty while retaining a critic-free design.The substitution is presented as a drop-in replacement for existing pipelines.
3.1. Clipping-Free Policy Optimization: The quadratic penalty produces an everywhere-differentiable objective with a continuous restoring force toward the trust region instead of zero gradients beyond a clipping boundary.The penalty is minimized at r_i,t = 1 and grows quadratically with deviation.
3.1. Clipping-Free Policy Optimization: CFPO modifies only the surrogate objective, making it agnostic to how advantages are estimated.The full objective includes KL regularization against a reference policy through coefficient β.
3.2. Advantage Estimation: For reasoning tasks, group-relative advantages normalize response rewards within sampled groups, supporting learning when reward magnitudes vary across problems.The estimator is suited to verifiable rewards such as mathematical correctness.
3.2. Advantage Estimation: For alignment tasks, RLOO uses the remaining K −1 samples as an unbiased, parameter-free baseline and does not normalize by standard deviation.This produces different gradient scaling behavior from group-relative advantages.
3.2. Advantage Estimation: CFPO shows consistent stability benefits with both advantage estimators, suggesting benefits across differing advantage-computation choices.The reported comparison attributes the stability pattern to replacing clipping with a convex penalty rather than to one estimator alone.

4. Experimental Setup

The experiments compare CFPO with clipped-policy baselines across reasoning and alignment settings, varying model families, frameworks, reuse iterations, and evaluation benchmarks. Training dynamics are examined under off-policy mechanisms, with CFPO contrasted against GRPO in reasoning and RLOO in RLHF.

4. Experimental Setup: Reasoning experiments train GRPO and CFPO on 7,500 MATH training problems using Qwen2.5-1.5B, 3B, and 7B base and Instruct models.The setup uses Open-R1/TRL, with some ablations using verl and β = 0.0 unless otherwise specified.
4. Experimental Setup: The framework comparison distinguishes sample reuse from mini-batch gradient updates: TRL supports sample reuse, whereas verl follows the standard deep-RL off-policy structure.This implementation difference affects when clipping may be observed.
4. Experimental Setup: The study evaluates 48 TRL models across iterations, model sizes, policy losses, and model bases, plus 32 verl models across iterations, batch ratios, and policy losses.TRL iterations extend to 16, while verl iterations extend to 8.
4. Experimental Setup: Reasoning evaluation covers MATH500, GSM8K, AIME24, and GPQA-Diamond using sampling-based decoding with temperature 0.6 and top-p 0.95.The same chat-style prompting format used in training is applied during evaluation.
4. Experimental Setup: RLHF experiments compare CFPO with RLOO using Llama-3 backbones, OpenRLHF SFT and reward models, default hyperparameters, and k = 2 rollouts per prompt.KL-free settings explicitly set the KL coefficient to zero.
4.2. RLHF: Under increasing sample reuse, GRPO gains reward faster initially but develops rising clip ratios and instability, whereas CFPO maintains low clip ratios and stable training before reaching comparable reward.The comparison uses reward trajectories on top and clip-ratio trajectories below across increasing iterations per update.
4. Experimental Setup: RLHF evaluation includes AlpacaEval 2.0, Arena-Hard v0.1, MT-Bench, IFEval, and selected OpenLLM Leaderboard capability tasks.AlpacaEval 2.0 and Arena-Hard use GPT-4.1 judging, while MT-Bench uses GPT-4.

5. Results and Analysis

Across reasoning and alignment experiments, CFPO trades faster early optimization for greater stability while preserving competitive downstream performance. It extends stable reasoning training, reduces verbosity exploitation and capability degradation in RLHF, and remains robust across off-policy settings.

Reasoning: CFPO progresses more gradually than GRPO, which achieves faster early reward gains but becomes increasingly unstable as iteration counts grow.This trade-off reflects CFPO’s more conservative optimization dynamics and sustained progress during longer training.
Reasoning: CFPO remains stable across instruction-tuned model scales and training durations, while GRPO exhibits instability at higher iteration counts.The same stability pattern appears in both cold-start and instruction-tuned RLVR experiments.
Off-policy training: Iteration-based sample reuse more readily amplifies policy lag than batch-ratio off-policy updates with fresh data, although accumulated off-policy pressure can still induce collapse.CFPO maintains lower clipping ratios, slower entropy decay, and more stable updates across these configurations.
RLHF: In RLHF, CFPO produces steadier reward improvement, stable response lengths, and lower clipping ratios, whereas RLOO’s aggressive optimization increases verbosity and clipping activity.The comparison uses Llama3-8B with RLOO advantage estimation and changes only the policy objective.
RLHF: CFPO outperforms RLOO by 3–4 percentage points on length-controlled AlpacaEval while remaining competitive on Arena-Hard and MT-Bench.The gap indicates that RLOO’s similar raw win rates are sensitive to verbosity, whereas CFPO’s scores are more consistent after length control.
RLHF: CFPO reduces alignment tax from 12–16% under RLOO to 4–5% while preserving more capabilities across ARC, HellaSwag, and Winogrande.CFPO’s conservative updates are associated with less capability degradation after RLHF.

6. Related Work

Prior work established trust-region and clipping-based approaches for stable policy optimization, while later methods addressed their limitations in language-model reinforcement learning. CFPO instead removes clipping and uses a quadratic penalty as a unified alternative across RLHF and reasoning.

Stable Policy Gradient Methods: Trust-region methods underpin stable policy optimization, while PPO made such methods practical through clipped objectives.PPO clipping became dominant in both continuous-control and language-model training, despite evidence that it does not bound KL divergence.
Reinforcement Learning for Language Models: RLHF uses preference-trained reward models for alignment, while DPO and related methods bypass reward models to address reward overoptimization.The related-work landscape includes IPO, KTO, and SimPO as preference-optimization variants.
Reinforcement Learning for Language Models: GRPO standardizes verifiable-reward reasoning with group-relative advantages, but clipping-related instabilities have motivated methods such as DAPO and Dr.GRPO.These approaches modify clipping behavior or token weighting rather than replacing clipping entirely.
Reinforcement Learning for Language Models: CFPO replaces clipping entirely with a quadratic penalty, providing one approach across RLHF and reasoning settings.This distinguishes CFPO from methods that patch or modify clipping behavior.

7. Discussion and Future Work

The evaluation covers 1.5B–8B Qwen and LLaMA models on MATH and OpenRLHF-related settings, leaving broader scale, architecture, data, and domain coverage for future work.

Scope limitations: The experiments are limited to Qwen and LLaMA families trained on MATH and OpenRLHF datasets.They do not evaluate frontier-scale models, substantially longer training horizons, broader architectures, or domains with sparse or noisy rewards.
Future work: Whether CFPO’s conservative updates remain beneficial or become overly restrictive at frontier scale is unresolved.The paper identifies larger models, longer horizons, and applications such as code generation or agentic tasks as future directions.

8. Conclusion

CFPO replaces clipped PPO and GRPO objectives with smooth quadratic regularization derived from Total Variation constraints. Across reasoning and alignment settings, it improves training stability while maintaining competitive downstream performance without extra computational cost or hyperparameters.

CFPO replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, providing smooth gradients and implicit trust-region bounds.
Across reasoning and alignment experiments, CFPO substantially delays collapse at high iteration counts while maintaining gradual entropy consumption and reducing alignment tax.
CFPO achieves competitive downstream performance at no additional computational cost, requiring only a one-line code change.
The results position CFPO as a promising drop-in alternative to clipped objectives for language-model post-training.

A.3. The ϵ-Aligned Objective Class

The ϵ-aligned objective class converts a constrained probability-ratio problem into an unconstrained objective whose optimum satisfies the desired boundary. SPO meets this criterion because it is differentiable and convex, whereas PPO clipping creates zero-gradient regions.

The constrained optimization problem uses rt(θ) = πθ(at|st)/πθold(at|st) and the estimated advantage ˆAt to optimize policy updates.
For a single nonzero advantage, the linear objective reaches its optimum at the constraint boundary r∗ = 1 + sign(A) · ϵ.
An ϵ-aligned objective is differentiable and convex in r and maximized at r = 1 + sign(A) · ϵ.
SPO is ϵ-aligned because its quadratic objective is differentiable and convex, with its stationary point at the desired boundary.
PPO is not ϵ-aligned because clipping makes gradients vanish outside the clipping range, eliminating corrective signals for those data points.

B. Training Details

The reasoning and RLHF experiments use specified model, prompt, framework, and hyperparameter configurations, with defaults retained except where policy loss or KL settings are explicitly modified.

Reasoning Hyperparameters: Reasoning experiments use the same listed training hyperparameters for base and instruct Qwen2.5 model variants.
Reasoning Hyperparameters: Table 3 lists only hyperparameters affecting the learned policy or evaluation, while unspecified fields inherit TRL v0.8 defaults.
Reasoning Hyperparameters: Reasoning training uses shared system prompts that request step-by-step solutions and boxed final answers, with prompt variants specified by model scale.
RLHF Hyperparameters: RLHF experiments use OpenRLHF with default hyperparameters, modifying only the policy loss to use CFPO and setting the KL coefficient to zero in KL-free settings.
RLHF Hyperparameters: RLHF uses Llama-3-8B-SFT-Mixture, Llama-3-8B-RM-700K, and the OpenRLHF prompt-collection-v0.1 dataset.

C. Qwen2.5 Results in TRL

Across Qwen2.5 reasoning evaluations, GRPO and CFPO generally achieve similar downstream behavior, but CFPO is more robust as iteration counts and off-policy pressure increase. GRPO’s faster early gains are accompanied by clipping, entropy reduction, and instability, whereas CFPO progresses more gradually and remains stable longer.

Instruction-Tuned RLVR: Both methods achieve similar downstream reasoning behavior on instruction-tuned Qwen2.5 models, while CFPO remains stable across iteration counts and GRPO degrades as off-policy pressure increases.
Cold-Start RL: CFPO maintains robust performance across cold-start Qwen2.5 scales, whereas GRPO degrades or collapses as iteration counts grow.
Training Dynamics: GRPO exhibits faster early reward improvement, while CFPO progresses more gradually and converges later during cold-start training.
Training Dynamics: Higher iteration counts primarily induce instability, while increasing batch ratio alone remains comparatively stable and provides no systematic validation-reward gains.
Training Dynamics: GRPO clipping activity rises with batch ratio and iteration count, with sharp clipping increases preceding instability; CFPO maintains lower, more stable clipping behavior.
Training Dynamics: GRPO and CFPO show similar KL magnitudes, indicating that their stability difference is associated with update regularization rather than large inter-policy shifts.
Training Dynamics: GRPO reduces entropy faster, especially at higher iteration counts, whereas CFPO retains higher entropy through more gradual policy concentration.