Source-linked AI summary
When KL Regularization Misfires in Group Policy Optimization
Fei Ding
TL;DR
Removing reference-policy KL can sometimes improve group policy optimization, raising the question of how reference information should enter group-relative updates. The paper analyzes seven KL–reward failure modes and proposes ZCPO, which calibrates within-group reward coefficients using conditional KL. In the reported settings, ZCPO achieves higher mean task accuracy than several comparison methods.
Problem
The paper asks how reference-policy information should enter group-relative optimization, motivated by observations that removing reference KL can help.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The paper’s mathematical reasoning and long-context experiments support ZCPO’s design in the evaluated settings.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Why does removing reference-policy KL regularization sometimes improve group policy optimization? This motivates studying how reference-policy information should enter group-relative updates. We analyze seven potential failure modes in the interactions between KL and rewards: residual KL updates after reward clipping, after gradient cancellation, and in groups with identical rewards; KL growth with response length and an imbalance in its relative contribution; KL concentration on a small number of tokens; and sampling noise when k1 is incorporated into rewards. We propose Zero-Sum Calibrated Policy Optimization (ZCPO), which uses relative drift measured by conditional KL to calibrate within-group reward coefficients and integrates them into the base surrogate. Mathematical reasoning experiments and ablations support this design's effectiveness in our settings.
1 INTRODUCTION
The observation that removing reference KL can help group-relative optimization raises the question of how reference-policy information should enter its updates. The paper analyzes seven potential KL–reward failure modes and proposes ZCPO, which calibrates group-relative reward coefficients using conditional KL.
1 INTRODUCTION
- Removing reference KL increased PAPO’s multimodal reasoning avg@8 from 47.92% to 50.18% for Qwen2.5-VL-3B and from 58.78% to 61.30% for Qwen2.5-VL-7B (Wang et al., 2026b, Appendix F, Table 4).The introduction also notes that PPO without KL outperformed variants adding KL to the loss or reward in Open-Reasoner-Zero text reasoning ablations (Hu et al., 2025).Source: §1 · p. 1
- Prior work’s observation that removing reference KL can help motivates asking how reference-policy information should enter group-relative optimization.The paper frames its analysis around the interaction between independent KL and reward updates, and within-group calibration using reference-policy drift.Source: §1 · p. 1
- The paper identifies seven potential KL–reward failure modes and proposes ZCPO, which uses conditional KL to calibrate group-relative reward coefficients.The authors report that mathematical reasoning experiments and ablations support the design in their settings.Source: §1 · p. 1
2 RELATED WORK
Related work spans changes to group-relative optimization, analyses of KL placement and estimation, and methods that adjust constraints or recover learning signals. The paper distinguishes its focus on interactions between KL and group-relative reward updates from these approaches.
2 RELATED WORK
- Related group-relative optimization methods modify clipping, normalization, aggregation, or trust regions, while the paper focuses on how KL interacts with group-relative reward updates.Examples include DAPO, GSPO, and GMPO; the passage also names methods improving clipping or trust regions (Xi et al., 2026; Petrenko et al., 2026; Lee et al., 2026; Wang et al., 2026a; Cai et al., 2026).Source: §2 · pp. 1–2
- Prior analyses study stationary policies under joint reward normalization and reference penalties, KL-gradient estimation bias, and k1/k3 gradient bias in rewards or losses (Tang & Munos, 2025; Shah et al., 2026).The paper distinguishes its focus on interactions with reward clipping and group filtering, and on conditional KL as a within-group zero-sum correction coefficient.Source: §2 · p. 2
- GVPO and RPG refresh the reference policy during training, which may alleviate constraints from a fixed initial reference, but sampled log-ratios can retain conditional sampling fluctuations (Zhang et al., 2025; 2026).Within-group centering alone also does not guarantee that those fluctuations disappear.Source: §2 · p. 2
- Per-prefix conditional KL has been used to reduce estimation variance, while reference-policy entropy weighting relaxes constraints at uncertain positions (Amini et al., 2025; Vassoyan et al., 2025).The paper says these works do not study its joint design of length normalization, within-group centering, and reward-active-group gating, or address overconfident reference positions, respectively.Source: §2 · p. 3
3 SEVEN POTENTIAL FAILURE MODES IN THE INTERACTION BETWEEN KL AND REWARDS IN GRPO AND ITS VARIANTS
The section analyzes seven potential mismatches between independent KL signals and group-relative reward updates, under specified conditions. They concern residual KL updates, length and aggregation effects, token concentration, and sampling noise.
3 SEVEN POTENTIAL FAILURE MODES IN THE INTERACTION BETWEEN KL AND REWARDS IN GRPO AND ITS VARIANTS
- When reward clipping makes the base surrogate locally flat, its reward gradient vanishes but the independent KL gradient remains, so the token’s direct update comes from KL.This is the residual-update case after reward clipping.Source: §3 · pp. 3–4
- When within-group reward gradients cancel during aggregation, independent KL gradients need not cancel; shared contexts and identical sampled tokens give an exact cancellation case under open gates and equal weights.Approximate cancellation for identical segments in different contexts depends on their actual gradients and reward coefficients.Source: §3 · p. 4
- In a participating group with identical rewards, all normalized advantages are zero, leaving only the independent KL term in the group’s direct gradient.That KL contribution may remain nonzero.Source: §3 · p. 4
- Without length normalization, nonnegative per-token conditional KL accumulates across a response, so longer responses incur larger penalties when per-token KL values are comparable and positive.The independent KL loss can also affect response length through the gradient at the stopping position.Source: §3 · p. 4
- As more tokens are aggregated, KL’s relative gradient contribution may increase if reward-gradient cancellation slows reward-gradient growth while KL gradients accumulate faster.This is a potential imbalance between the two branches, not a claim that it occurs in every setting.Source: §3 · p. 4
- When KL signals concentrate on a few tokens, their pull toward the reference may weaken conflicting reward updates at those positions and potentially suppress beneficial policy changes.The passage uses the per-token k3 KL loss as an example.Source: §3 · p. 4
- With k1 coefficients, token sampling can create response-level KL coefficient noise even when responses share the same length-normalized conditional KL, and within-group centering need not remove it.For the stated single-token example, all responses share the same context and differences arise from sampling.Source: §3 · p. 4
4 ZERO-SUM CALIBRATED POLICY OPTIMIZATION (ZCPO)
ZCPO uses conditional KL to calibrate response-level coefficients within a group, then substitutes the coefficients into the base surrogate. Its construction targets zero-sum updates that stay close to normalized rewards while favoring lower drift.
4 ZERO-SUM CALIBRATED POLICY OPTIMIZATION (ZCPO)
- ZCPO calibrates group-relative update coefficients using conditional KL and incorporates them into the base algorithm’s surrogate.The coefficients are computed from within-group reward advantages and drift; groups are gated by group size and nonzero reward spread.Source: §4 · p. 4, Figure 2 · p. 5
- The calibrated coefficients are held fixed during the backward pass and replace the reward coefficient in the base surrogate, whose clipping and aggregation rules remain supplied by the base algorithm.Gradients propagate through probability ratios, not through the coefficients themselves.Source: §4 · p. 5
- The calibration keeps coefficients close to normalized reward coefficients while favoring lower drift, subject to a within-group zero-sum constraint.A strictly convex objective yields the unique optimum for fixed inputs under that constraint.Source: §4 · pp. 5–6
- For equal-reward responses, lower mean KL yields larger joint coefficients when β > 0; adding a shared within-group KL constant leaves the calibrated coefficients unchanged.Conditional KL removes current-token sampling noise given a prefix, but variation from different sampled prefixes remains.Source: §4 · p. 6
- Zero-sum coefficients do not ensure reward increases or KL decreases at every step; for Ai > 0 and βKi > Ai/sq, Ci can still be negative.The KL branch may remain nonzero when reward gradients cancel, and the group gate skips identical-reward groups.Source: §4 · p. 6
5 EXPERIMENTS
The experiments compare ZCPO with DAPO-based baselines and test its behavior across KL–reward diagnostics and component ablations. In the tested setting, ZCPO achieves higher mean task accuracy than several comparison methods.
Ablation analysis
- Removing within-group KL centering causes the largest mean-accuracy drop; removing other listed components also lowers mean accuracy.The ablations evaluate combined effects because a component can affect multiple update mechanisms.Source: §5 · p. 6
Experimental analysis
- ZCPO achieves higher mean task accuracy than DAPO, RPG, GVPO, and GOPO in the reported setting.Table 1 reports avg@32 (%) for AIME24 and AIME25, with means over three random seeds.Source: §5 · p. 6, Table 1 · p. 7
F1
- At clipped tokens, vanilla retains KL gradients, with a 68–172× KL-gradient overallocation metric; ZCPO’s direct branch gradients are zero where its gate closes.Disabling KL at reward-clipped positions yields higher mean accuracy than the independent-KL baseline and the all-token norm control.Source: §5 · p. 7, Figure 3 · p. 8
F4 and F5
- At β = 4, vanilla’s mean response length is 611 tokens at step 100, versus 1695 for ZCPO; vanilla length also decreases as β increases.Vanilla’s median KL share in longer groups is 12 times that in shorter groups; ZCPO’s short/long medians differ by 0.24 percentage points.Source: Figure 3 · p. 8, §5 · pp. 8–9
F2
- Across high- and low-cancellation groups, vanilla’s KL gradient share differs by about 524×, versus about 3.45× for ZCPO.Reward-branch gradient cancellation reaches 99.6% during training under the stated estimator.Source: §5 · p. 8, Figure 3 · p. 8
F6
- A small fraction of tokens accounts for most of the KL signal, while within-group centering removes shared length-normalized KL across responses.At step 20, the first 10 tokens contribute approximately 4% of the variance of K; ZCPO shows greater initial-token drift than vanilla in this setting.Source: Figure 3 · p. 8, §5 · p. 9
F7
- At steps 3–10, 42%–59% of k1-control responses receive negative KL estimates before centering, compared with 0 for ZCPO.The control’s within-group dispersion is close to the small-KL sampling-noise reference scale; conditional KL removes current-token sampling noise given its prefix, but variation from sampled prefixes remains.Source: Figure 3 · p. 8, §5 · p. 9
6 CONCLUSION AND LIMITATIONS
The paper studies seven potential KL–reward failure modes and proposes ZCPO, using conditional-KL relative drift to calibrate within-group zero-sum coefficients. Experiments with Qwen models support the design in the evaluated settings; the supplementary material includes reproducibility code.
6 CONCLUSION AND LIMITATIONS
- The conclusion summarizes seven potential KL–reward failure modes and ZCPO’s conditional-KL calibration of within-group zero-sum coefficients.The reported support comes from mathematical reasoning experiments and long-context experiments, with long-context ablations evaluating group gating.Source: §6 · p. 9
AI USE STATEMENT
- The paper states that no AI was used.Source: AI USE STATEMENT · p. 10
REPRODUCIBILITY STATEMENT
- The supplementary material provides code that can be run to reproduce the results.Source: REPRODUCIBILITY STATEMENT · p. 10
A EXPERIMENTAL CONFIGURATION
The main experiments use a DAPO-based configuration, with settings and KL coefficients listed in Table 2. The appendix describes checkpoint selection, sampling and filtering, and numerical implementation details for ZCPO.
A EXPERIMENTAL CONFIGURATION
- The base training configuration follows DAPO, and Table 2 lists training and evaluation settings and main-experiment KL coefficients.Source: A EXPERIMENTAL CONFIGURATION · p. 13, Table 2 · p. 13
Checkpoint selection
- Main-experiment checkpoints are selected at matching cumulative GPU compute times, independently of AIME24 and AIME25 evaluation results.Source: A EXPERIMENTAL CONFIGURATION · p. 13
Computation reuse
- ZCPO reuses policy forward-pass outputs and logits, while fp32 projection recomputation and KL reduction add approximately 0.5% GPU compute time.The fp32 recomputation and reduction address rounding errors in bf16 logits when the conditional distributions are close.Source: A EXPERIMENTAL CONFIGURATION · p. 13
Sampling and filtering
- Dynamic sampling filters groups with all-correct or all-incorrect answers, while overlong filtering masks truncated-response losses.The mathematical reasoning experiments set πold ← πθ before each rollout; a length penalty is added to the rule-based correctness reward.Source: A EXPERIMENTAL CONFIGURATION · pp. 13–14
B SENSITIVITY TO β
The sensitivity comparison separates the effect of reward normalization from KL-coefficient calibration, while diagnostics examine how KL contributes to gradient norms and conditional sampling noise.
B SENSITIVITY TO β
- KL-coefficient calibration increases mean AIME24 accuracy by 8.1 and 8.7 percentage points under the two reward-normalization schemes.Changing to maximum-absolute-advantage normalization alone increases mean accuracy by 0.8 points with calibration disabled and 1.4 points with it enabled.Source: B SENSITIVITY TO β · p. 15, Table 4 · p. 15
C DIAGNOSTICS OF THE KL GRADIENT SHARE
- Within each method, the reward-branch gradient norm remains essentially unchanged across β, while the KL-branch norm grows approximately linearly with β.The methods' absolute norms are on different scales, so the passage recommends comparing them only within a method.Source: C DIAGNOSTICS OF THE KL GRADIENT SHARE · p. 16
- ZCPO’s zero conditional sampling error applies to the local estimation term for a given prefix; variation across sampled prefixes remains.Figure 4 contrasts the k1 coefficient's residual RMSE with ZCPO’s zero local residual and shows GVPO training curves alongside the measured sampling residual.Source: Figure 4 · p. 17
D OPTIMALITY OF ZERO-SUM COEFFICIENT CALIBRATION
For fixed inputs in an active group, the section characterizes zero-sum coefficient calibration as a constrained optimization problem. Its solution is unique and favors lower-drift responses when rewards are equal.
D OPTIMALITY OF ZERO-SUM COEFFICIENT CALIBRATION
- The coefficients are calibrated for fixed samples, rewards, and current policy, with a zero-sum constraint and a Lagrange multiplier.The coefficients may be positive or negative; the inputs Ai/sq and eDi are treated as constants.Source: D OPTIMALITY OF ZERO-SUM COEFFICIENT… · p. 18
- The solution satisfies the zero-sum constraint and is the unique global optimum because the objective is strictly convex under that constraint.The Hessian with respect to the coefficients is the identity matrix.Source: D OPTIMALITY OF ZERO-SUM COEFFICIENT… · p. 18
- Adding a shared constant to every eDi leaves the optimal coefficients unchanged, and for equal rewards lower eDi yields a larger coefficient when β > 0.The equal-reward ordering is stated for two responses in the same active group.Source: D OPTIMALITY OF ZERO-SUM COEFFICIENT… · p. 18
- This optimality result concerns coefficient calibration with fixed inputs; the drift score is linear, not a policy KL divergence, and the policy parameters are fixed.During training, the detached coefficients are inserted into the base surrogate objective.Source: D OPTIMALITY OF ZERO-SUM COEFFICIENT… · p. 18
E CROSS-MODEL AND CROSS-TASK EXPERIMENTS
The long-context experiments compare GRPO variants, including ZCPO and a group-gating ablation, on a fixed GoLongRL subset and six benchmarks. The appendices document their training and configuration settings.
E CROSS-MODEL AND CROSS-TASK EXPERIMENTS
- Starting from Qwen3-4B-Thinking-2507, the long-context experiment compares standard-KL GRPO, no-KL GRPO, GRPO-ZCPO, and ZCPO without group gating on six benchmarks.The methods use a fixed subset of 8,000 GoLongRL examples; standard KL uses coefficient 0.001, and ZCPO and its ungated ablation use β = 4.Source: E CROSS-MODEL AND CROSS-TASK EXPERIMENTS · p. 19, Table 6 · p. 19
E.2 ADDITIONAL CONFIGURATION FOR LONG-CONTEXT EXPERIMENTS
- The appendix provides long-context training hyperparameters and additional model, KL, sampling, parallelism, checkpoint, and validation settings.The configuration includes checkpoint retention and validation frequency, as well as actor/reference and rollout settings.Source: §E.1 · p. 20, Table 7 · p. 20, §E.2 · pp. 21–22
E.1 TRAINING HYPERPARAMETERS FOR LONG-CONTEXT EXPERIMENTS
- The evaluation tasks are DocMath, LongBench-V2, Frames, MRCR, CorpusQA, and LongBench-v1 QA, whose five subsets are listed separately.The passage identifies the subsets as 2WikiMultihopQA, HotpotQA, MuSiQue, NarrativeQA, and Qasper.Source: §E.1 · p. 20
F CROSS-MODEL GENERALIZATION: GLM-4-9B-0414 MATH EXPERIMENTS
The paper reports that ZCPO improves AIME24 mean accuracy over DAPO without KL on a GLM base, offering preliminary cross-family support. It also constructs cases where independent KL reverses weak reward contributions and where global β tuning entails a trade-off.
F CROSS-MODEL GENERALIZATION: GLM-4-9B-0414 MATH EXPERIMENTS
- On GLM-4-9B-0414, adding ZCPO improves AIME24 mean accuracy over DAPO without KL, providing preliminary support for cross-family generalization.The comparison uses AIME24 avg@32 and follows the main experiments' training and evaluation settings apart from the training start.Source: F CROSS-MODEL GENERALIZATION:… · p. 23, Table 9 · p. 23
G INDEPENDENT KL SUPPRESSION OF WEAK REWARD SIGNALS
- With τ = 10^-3, εadv = 10^-6, and β = 0.04, the combined coefficient is approximately −0.0189649, reversing the contribution for a positive-reward response.The construction uses current and old probabilities of 0.4 and a reference probability of 0.2.Source: G INDEPENDENT KL SUPPRESSION OF WEAK… · p. 24
- For the negative-reward response, the same construction gives a positive update coefficient of approximately +0.0122982; both reversals are unclipped token-level constructions, not training-performance results.The global-β example also shows that protecting a strongly conflicting position can constrain regularization at a weaker one.Source: G INDEPENDENT KL SUPPRESSION OF WEAK… · p. 24
- When relative KL contributions concentrate at a few reward-conflicting positions, reducing global β weakens regularization elsewhere without changing the disparity between positions.In the stated two-position construction, λhigh is 100 times λlow.Source: G INDEPENDENT KL SUPPRESSION OF WEAK… · p. 24
G.1 HOW ZCPO MITIGATES THIS FAILURE MODE
ZCPO centers conditional-KL drift within each group and normalizes reward advantages to calibrate relative coefficients. The section also specifies frozen-coefficient mini-batch updates and limits what its analyses establish about task performance.
G.1 HOW ZCPO MITIGATES THIS FAILURE MODE
- ZCPO removes shared drift from the coefficient penalty while retaining relative drift differences; with equal rewards, lower-drift responses receive larger coefficients.This turns reference information into a relative preference rather than a direct penalty for common policy deviation.Source: §G.1 · p. 25
- For β = 0.04 and τ = 10^-3, the weak-positive and weak-negative coefficients are approximately -0.0189649 and +0.0122982 under ordinary k3, versus +0.001 and -0.001 under ZCPO.The example uses 16 single-token responses and a shared conditional KL; removing shared drift restores both reward directions without reducing global β.Source: §G.1 · p. 25
- Maximum-absolute-advantage normalization fixes the strongest reward signal, so relative KL calibration varies with β and within-group drift differences.Unlike standard-deviation normalization, it fixes the peak scale and removes reward-peak variation as a source of calibration fluctuation.Source: §G.1 · pp. 25–26
- When a dominant reward factor becomes shared, ZCPO cancels it and restores the strongest remaining reward difference to unit magnitude.Relative KL also removes shared drift, allowing residual differences to regain relative weight without an additional curriculum scheduler.Source: §G.1 · p. 26
H MINI-BATCH UPDATES WITH FROZEN COEFFICIENTS
- ZCPO computes coefficients at the sampling-policy snapshot and keeps rewards, groups, and coefficients fixed through that rollout’s mini-batch updates.The identical-reward gate skips inactive groups; the clipping gate can still change as current probability ratios change.Source: H MINI-BATCH UPDATES WITH FROZEN… · p. 27
I EVIDENCE SCOPE AND DISCUSSION
- The analyses characterize specified interactions and mechanisms, while mathematical reasoning, long-context, and cross-model experiments support full-method task performance only in evaluated configurations.The contribution of individual mechanisms to task gains is assessed separately through analyses and controls under their respective conditions.Source: I EVIDENCE SCOPE AND DISCUSSION · p. 27