Source-linked AI summary
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li, Deng Cai, Min Chen, Hao Zhang
TL;DR
Entropy-based advantage shaping cannot distinguish useful uncertainty from confusion in RLVR. CPO uses reference-guided contrastive disagreement to provide correctness-aware token-level signals, substantially outperforming entropy-based methods while preserving out-of-domain generalization.
Problem
Entropy-based RLVR advantage shaping cannot distinguish useful uncertainty from detrimental confusion or assess individual token contributions to correctness.
Method
CPO shapes advantages using token-level contrastive disagreement between reference-guided and vanilla generation distributions as a correctness signal.
Results
CPO outperforms GRPO by 7.7% and 8.5% on two backbones while maintaining or improving out-of-domain generalization.
Takeaways & Limitations
Balancing correct-response exploration with incorrect-response exploitation supports stronger reasoning performance and generalization.
Takeaways & Limitations
CPO relies on reference answers to construct its correctness-informed distribution, limiting applicability to settings with verifiable rewards.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.
1 Introduction
The introduction identifies entropy as an unreliable correctness signal in RLVR and proposes Contrastive Policy Optimization (CPO), which uses reference-guided disagreement to shape advantages. It presents CPO as theoretically connected to on-policy distillation and empirically stronger than entropy-based methods while preserving out-of-domain performance.
- Background: RLVR improves reasoning in mathematics and programming, while GRPO estimates advantages by sampling multiple outputs and normalizing rewards within each group.GRPO removes the need for a separate critic model.
- Problem: Entropy-based advantage shaping improves exploration at uncertain tokens but cannot reliably distinguish valuable exploration from detrimental errors.Existing methods consequently disagree over whether to favor high- or low-entropy tokens, motivating a correctness-aware signal.
- Proposed Method: CPO quantifies token-level contrastive disagreement between reference-guided and vanilla generation distributions to provide correctness-aware advantage shaping.The paper reports theoretical and empirical evidence that this disagreement reliably indicates token-level correctness.
- Relation to Prior Work: Contrastive disagreement provides a theoretical grounding for on-policy distillation, with OPD’s reverse KL appearing as a special case.The framework also covers teachers based on external models, reference-conditioned self-teachers, critic feedback, and other contextual signals.
- Results: 7.7% and 8.5%: CPO outperforms GRPO on average on Qwen2.5-Math-7B and Qwen3-Base-4B, respectively, while largely preserving out-of-domain performance.The analysis attributes the gain to CPO’s focus on discriminative correctness features rather than the linguistic variability emphasized by entropy methods.
2 Contrastive Policy Optimization
Contrastive Policy Optimization uses disagreement between vanilla and correctness-informed token distributions as a correctness-aligned signal for token-level advantage shaping. Its theoretical interpretation and empirical validation support distinguishing incorrect-leaning tokens and enriching policy gradients, including when trajectory advantages are zero.
- Correctness-informed distribution: CPO contrasts prior and correctness-conditioned next-token distributions, with the ideal posterior defined as πpost(yt | x, y<t) = P(Yt = yt | X = x, Y<t = y<t, C).The posterior represents token probabilities conditioned on eventual correctness.
- Correctness-informed distribution: For a fixed prefix, the log-ratio increases monotonically with token correctness probability: below-average tokens have negative values, while above-average tokens have positive values.The correctness probability is defined over future continuations, so the signal applies whether the sampled rollout is correct or incorrect.
- Correctness-informed distribution: Reference-conditioned generation provides a practical proxy for the unavailable ideal posterior because the reference encodes ground truth and prompts the model to refine its response.This proxy is treated as approximately equivalent to conditioning on the correctness event.
- Empirical validation: Accuracy@16 validation shows that increasingly negative disagreement lowers accuracy from 57% at exp(δ) = 1 to approximately 48% at exp(δ) = 0.1, identifying incorrect-leaning tokens.Positive disagreement maintains stable accuracy near baseline, producing a clear asymmetry.
- Advantage shaping: CPO shapes trajectory advantages with token-level disagreement, using Ai,t + α · δi,t as a base form while preserving trajectory advantage as the primary determinant.The method also applies disagreement to zero-advantage responses, where it supplies learning signals for more informative gradients and exploration.
3 Experiment
Experiments across two backbones and in-domain and out-of-domain reasoning benchmarks show that CPO consistently outperforms GRPO, with especially strong gains on challenging long-horizon reasoning. Ablations and training analyses attribute these gains to reference-guided correctness signals, token-level shaping, and complementary exploration from correct responses and exploitation from incorrect responses.
- Training and evaluation: CPO is trained with GRPO on 7.5k MATH problems using Qwen2.5-Math-7B and Qwen3-Base-4B, then evaluated with Pass@16 across in-domain and out-of-domain benchmarks.Sampling uses temperature 0.6 and top-p 0.95; benchmarks include MATH500, AIME2024/2025, AMC23, and GPQA.
- Comparative results: CPO outperforms GRPO by 7.7% on Qwen2.5-Math-7B and 8.5% on Qwen3-Base-4B, with particularly notable gains on AIME2025.AIME2025 requires high per-step accuracy over long reasoning chains, making it a demanding test of fine-grained supervision.
- Data effectiveness: Applying CPO only to incorrect responses preserves most in-domain performance but reduces out-of-domain generalization by −4.2% on MMLU-PRO, while correct responses support generalization.The ablation indicates that negative examples target model weaknesses, whereas positive examples help maintain out-of-domain performance.
- Design ablations: The default reference-guided prompt works best, and CPO-πteacher(·|ref)) achieves the strongest results by combining teacher generalization with reference-based correctness signals.The prompt ablation compares the default single-turn format with 2-turn gold-shot and 2-turn 1-shot alternatives.
- Advantage design and training dynamics: Token-level disagreement is more robust than trajectory-level averaging, while CPO outperforms RL-ZVP by +3.2% average on zero-advantage prompts.Token-level shaping preserves token-wise credit-assignment variations and supports broader output distributions; 76% of fork tokens overlap with entropy while CPO additionally identifies execution-oriented tokens.
4 Related Work
Related work has incorporated entropy into RLVR to promote exploration, while other approaches develop fine-grained correctness signals to mitigate sparse rewards. These methods respectively face limitations in correctness specificity, annotation cost, or computational practicality.
- Entropy-Intervened RL: Entropy-based RLVR builds on entropy’s use for exploration and includes entropy regularization and bidirectional regularization interventions.Entropy was introduced to encourage exploration and prevent premature convergence; later work analyzed training dynamics and added entropy terms to RLVR objectives.
- Fine-grained Correctness Signals: Fine-grained correctness methods address sparse RLVR rewards through human-labeled step signals, automated tree search, and on-policy distillation.Human annotation is labor-intensive, tree-based search is computationally impractical because of repeated sampling, and on-policy distillation uses teacher feedback on self-generated outputs.
5 Conclusion
The paper introduces Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement as a correctness-aware advantage-shaping signal in RLVR. Theoretical and empirical results show that this signal indicates token-level correctness, improves challenging math reasoning, and preserves out-of-domain generalization.
- 5 Conclusion: CPO uses token-level contrastive disagreement between reference-guided and vanilla generation as a correctness-aware signal for advantage shaping in RLVR.The framework is designed to replace less informative uncertainty signals with contrastive disagreement.
- 5 Conclusion: Theoretical and empirical results demonstrate that contrastive disagreement reliably indicates token-level correctness.
- 5 Conclusion: CPO significantly improves performance on challenging math reasoning tasks while maintaining out-of-domain generalization.
A Limitations · B Impact Statements
CPO’s main limitations are its added computational cost and dependence on reference answers for correctness-informed distributions, while its intended impact is improved credit assignment in high-accuracy domains alongside acknowledged risks from advanced models. The paper suggests asynchronous computation and alternative contextual signals as possible extensions.
- A Limitations: CPO requires an additional reference-guided forward pass to compute πpost, increasing training time by approximately 20% versus standard GRPO.This forward pass is identified as the primary computational overhead.
- A Limitations: Asynchronous computation could decouple the reference-guided pass from training, but slightly stale posterior estimates would introduce off-policy bias.The passage presents this as a promising mitigation with a stated tradeoff.
- A Limitations: CPO relies on reference answers to construct πpost, limiting applicability to settings with verifiable rewards.The passage frames reference-answer dependence as a current limitation of the method.
- A Limitations: Reference answers are only one instantiation of a correctness-informed distribution, motivating replacement with other contextual signals that do not require ground-truth answers.The passage suggests alternative contextual signals, including critic fe…
- B Impact Statements: The work’s primary goal is to improve credit-assignment accuracy during reinforcement learning.This goal defines the intended contribution and impact of the method.
- B Impact Statements: Improved credit assignment could benefit high-accuracy domains such as industrial workflow agents and scientific assistance.The passage identifies these application areas as potential social-impact domains.
- B Impact Statements: The authors acknowledge dual-use risks and potential amplification of biases present in training data.These risks are associated with advanced models and improved reasoning capabilities.
C Theoretical Analysis of Contrastive Generation Disagreement · D Generation Prompts
The theoretical analysis shows that contrastive generation disagreement tracks oracle-defined token correctness relative to prefix-average correctness, while the generation prompts provide the practical reference-guided distributions used in experiments. Together, these sections connect CPO’s correctness interpretation to its prompt-based implementation.
- C Theoretical Analysis of Contrastive Generation Disagreement: The oracle correctness event C is model-agnostic, while g(x, y<t, yt) measures the probability that choosing token yt eventually yields a correct completion.Correctness is defined relative to oracle ground truth, and g conditions on the prompt, prefix, and candidate token.
- C Theoretical Analysis of Contrastive Generation Disagreement: CPO compares reference-guided and vanilla token probabilities through δt(x, y) = log πpost(yt | x, y⋆(x), y<t) − log π(yt | x, y<t).The same tokens from a vanilla rollout are re-scored under a policy conditioned on the reference answer.
- C Theoretical Analysis of Contrastive Generation Disagreement: The ideal correctness-conditioned posterior is determined by the prior policy and oracle correctness event, with Bayes reweighting proportional to π(yt | x, y<t)g(x, y<t, yt).The prefix normalizer Z(x, y<t) is the prior-average correctness probability at that prefix.
- C Theoretical Analysis of Contrastive Generation Disagreement: The ideal log-ratio equals log correctness probability minus a prefix constant: log ˜πpost − log π = log g − log Z.This identity makes disagreement comparable across candidate tokens at a fixed prompt and prefix.
- C Theoretical Analysis of Contrastive Generation Disagreement: Under the reference-encoding assumption, practical δt serves as a correctness-aligned proxy, with large negative values indicating tokens suppressed by oracle-informed guidance and likely to be incorrect-leaning.Teacher-forced re-scoring empirically supports this approximation because large negative disagreements concentrate on incorrect trajectories and error-causing steps.
- C Theoretical Analysis of Contrastive Generation Disagreement: Negative ideal disagreement occurs exactly when a token’s correctness probability is below the prior-average correctness at that prefix, while positive disagreement occurs above it.The theorem establishes that ideal disagreement is strictly increasing in g(x, y<t, yt).
- D Generation Prompts: The generation-prompts section documents the vanilla and reference-guided prompts used in experiments, plus multi-turn gold-shot and multi-turn 1-shot ablation variants.It notes that prompt format is not CPO’s core contribution but affects how effectively reference-guided distributions shift toward correctness-oriented reasoning.
E Detailed training Implementations
CPO is evaluated with Qwen3-Base-4B and Qwen2.5-Math-7B on the 7.5k-problem MATH dataset. Training uses GRPO with specified rollout, sampling, context, optimization, and regularization settings, alongside reference-guided and multi-turn gold-shot prompts.
- Training setup: Experiments use Qwen3-Base-4B and Qwen2.5-Math-7B trained on the 7.5k-problem MATH dataset.The models represent general-purpose and domain-specific base models.
- Prompt design: Reference-guided generation asks for first-person, step-by-step reasoning from a question and reference answer, ending in \boxed{}.The prompt explicitly includes placeholders for both the question and reference answer.
- Prompt design: Multi-turn gold-shot generation supplies a question, reference answer, and a second request for step-by-step reasoning with the final answer in \boxed{}.The sequence uses alternating user and assistant turns before the final reasoning request.
- Training setup: CPO is implemented with GRPO using a prompt batch size of 1024, 8 rollouts per prompt, sampling temperature 1.0, and mini-batches of 256.These settings define the rollout and optimization batch structure during training.
F Additional Experiments and Analyses · F.1 Statistical Significance. · F.2 CPO on the Zero-Advantage Problem
The additional analyses validate CPO’s reliability across independent runs and explain its advantage on zero-advantage prompts. CPO combines stable training behavior with a correctness-aware signal that entropy-based methods lack.
- F.1 Statistical Significance.: CPO consistently outperforms GRPO and DAPO across benchmarks in three independent runs.Results are reported as mean ± std in Table 8.
- F.1 Statistical Significance.: CPO shows substantially lower variance than DAPO on challenging benchmarks such as AIME2024.The reported standard deviations are ±1.9 for CPO versus ±10.3 for DAPO.
- F.1 Statistical Significance.: Pass@16 reduces evaluation variance by aggregating 16 sampled responses per problem.This aggregation further reduces the influence of random sampling.
- F.2 CPO on the Zero-Advantage Problem: On zero-advantage prompts, RL-ZVP assigns high advantage to high-entropy tokens.This reflects the strategy of existing entropy-based methods under the controlled setting.
- F.2 CPO on the Zero-Advantage Problem: Entropy captures uncertainty without distinguishing tokens that support correct reasoning from those supporting incorrect reasoning.Consequently, high-entropy signals can reinforce confused or erroneous token choices.
- F.2 CPO on the Zero-Advantage Problem: CPO addresses the zero-advantage problem through a more principled token-level correctness-aware signal.The supplied passage introduces CPO’s token-level contrastive mechanism but does not provide the subsequent implementation details.
F.3 Comparison with SFT and Distillation Methods Table 10: Performance comparison of CPO with SFT and distillation baselines on Qwen2.5-Math-7B. Table 10: Performance comparison of CPO with SFT and distillation baselines on Qwen2.5-Math-7B.
On Qwen2.5-Math-7B, CPO outperforms vanilla SFT and distillation under equivalent data conditions. SFT substantially degrades average performance, distillation recovers part of it, and CPO achieves the strongest result.
- Experimental setup: Experiments compare CPO with SFT and distillation baselines under equivalent data conditions on Qwen2.5-Math-7B.The distillation baseline fine-tunes the model on trajectories self-sampled with reference-guided prompts.
- Performance comparison: -12.7 avg. is the degradation from vanilla SFT versus the original Qwen2.5-Math-7B.SFT suffers from distribution mismatch because concise ground-truth trajectories differ substantially from the model’s own generation distribution.
- Performance comparison: +3.5 avg. is the recovery achieved by distillation relative to the original Qwen2.5-Math-7B.Distillation mitigates the distribution mismatch by using reference-guided self-sampled trajectories.
- Performance comparison: +7.9 avg. is CPO’s improvement over the original Qwen2.5-Math-7B, making it the best-performing method by a significant margin.The comparison is reported across MATH-500, AIME2024, AIME2025, AMC2023, GPQA, MMLU-PRO, and KnowLogic, with an Avg. column.