PaperlayerView exact revision on arXiv

Source-linked AI summary

CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed, Rania Elbadry, Omar Fetouh, Fahad Shahbaz Khan, Salman Khan

arXiv:2605.19436v1cs.LGcs.CLcs.CV

TL;DR

RLVR assigns uniform token-level rewards, leaving decisive reasoning steps indistinguishable from filler. CEPO uses contrastive correct- and wrong-answer teachers to sharpen credit while preserving safety, and improves accuracy across five multimodal mathematical reasoning benchmarks.

  • Problem

    RLVR leaves the credit assignment problem unresolved because correct and incorrect trajectories assign uniform signals to all tokens, wasting gradient on filler.

  • Method

    CEPO replaces a single-reference evidence ratio with a contrastive ratio using correct- and wrong-answer teachers, with wrong answers drawn from rejected rollouts at no additional inference cost.

  • Results

    3.7% and 2.2% accuracy improvements over base at 2B and 4B scale across five multimodal mathematical reasoning benchmarks.

  • Takeaways & Limitations

    CEPO preserves RLSD’s structural safety guarantees while sharpening credit at decisive tokens and leaving filler unchanged, outperforming GRPO, OPSD, and SDPO across five benchmarks.

  • Takeaways & Limitations

    OPSD and SDPO fall below the untrained baseline on four of five benchmarks, indicating information leakage remains a practical failure mode for distribution-matching self-distillation.

Abstract

from arXiv · show

When a model produces a correct solution under reinforcement learning with verifiable rewards (RLVR), every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler. A natural fix is to condition the model on the correct answer as a teacher, identifying tokens it would have generated differently had it known the answer. Prior work shows this either corrupts training by leaking the answer into the gradient, or produces a weak signal that cannot distinguish decisive steps from filler, since both look equally surprising relative to the model's baseline. We propose Contrastive Evidence Policy Optimization (CEPO), which asks a sharper question at every token: not just "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?" A token satisfying both is a genuine reasoning step; one satisfying neither is filler. The wrong-answer teacher is constructed from rejected rollouts already in the training batch, incurring no additional sampling cost. We prove CEPO inherits all structural safety guarantees of the prior state of the art while strictly sharpening credit at decisive tokens, with the improvement vanishing exactly at filler positions. Empirically, CEPO achieves 43.43% and 60.56% average accuracy across five multimodal mathematical reasoning benchmarks at 2B and 4B scale, respectively, versus 41.17% and 57.43% for GRPO under identical training budgets. Distribution-matching self-distillation methods (OPSD, SDPO) fall below the untrained baseline, empirically confirming the information leakage our theory predicts. Our code is available at https://github.com/ahmedheakl/CEPO.

1 Introduction

RLVR methods assign uniform sequence-level credit, obscuring which reasoning tokens determine correctness. CEPO uses contrastive correct- and wrong-answer evidence to sharpen credit while preserving RLSD’s safety guarantees and improving benchmark accuracy.

  • Motivation: Uniform token credit leaves decisive reasoning steps indistinguishable from filler and can waste gradient signal in long reasoning chains.The paper links this bottleneck to slow convergence, noisy updates, and poor sample efficiency.
  • Motivation: Correct-answer self-distillation can leak answer information through vocabulary-wide gradients, corrupting training as the benign signal vanishes.The cited prior analysis attributes the harmful deviation’s variance to I(Y_t; R+ | X).
  • CEPO: CEPO contrasts correct- and wrong-answer teachers, using rejected rollouts already in the batch while canceling the student prior and reducing fluency confounding.Its Bayesian interpretation measures tokens that raise belief in the correct answer while lowering belief in the wrong answer.
  • CEPO: CEPO preserves RLSD’s direction anchoring and leakage-free gradients, and reduces exactly to RLSD when the wrong-answer teacher matches the student prior.The contrastive formulation strictly generalizes RLSD under the stated condition.
  • Results: CEPO’s sharper-credit conditions are validated as concentrating at semantically decisive positions rather than filler.The condition for correct trajectories is P−_T(y_t) < P_S(y_t).

2 Related Work

Prior RLVR credit-assignment methods trade off uniform sequence-level signals, privileged-information leakage, and limited token-level specificity. RLSD provides leak-free privileged credit but retains fluency and evidence-asymmetry limitations that motivate CEPO.

  • RLVR credit assignment: GRPO and related RLVR methods assign every token in a correct or incorrect trajectory the same sequence-level advantage.Token-level alternatives include Monte Carlo re-simulation methods such as VinePPO and SPO.
  • Privileged self-distillation: OPSD and SDPO use privileged correct-answer teachers, while SDPO additionally applies Jensen-Shannon divergence and EMA stabilization.HDPO applies the same general recipe to prompts where all rollouts fail.
  • Privileged self-distillation: Distribution-matching privileged teachers introduce vocabulary-wide r-conditioned gradients, a structural source of information leakage.The cited analysis reports OPSD and SDPO falling below the untrained baseline on four of five benchmarks.
  • RLSD: RLSD avoids leakage by evaluating the teacher ratio only at the sampled token under stop-gradient while anchoring its direction to the verifier.This makes RLSD both privileged-information using and leak-free without an auxiliary network.
  • RLSD: RLSD still conflates reasoning importance with base-rate fluency, provides indirect negative evidence, and cannot distinguish decisive steps from equally supported filler.These limitations concern signal quality rather than RLSD’s structural safety.

3 Method

CEPO replaces RLSD’s single-reference token evidence with a contrastive signal from correct and rejected rationales, while preserving verifier-anchored, leakage-free updates. Its contrastive weighting sharpens credit at decisive tokens, leaves filler near neutral, and recovers RLSD as a limiting case.

  • Contrastive Evidence Policy Optimization: CEPO partitions sampled rollouts into correct and wrong subsets, then conditions separate frozen teachers on sampled correct and rejected rationales.The resulting teacher distributions are compared with the student policy at each sampled token.
  • Background and RLSD: OPSD and SDPO use vocabulary-wide teacher-to-student divergence gradients that can encode the privileged correct rationale and produce information leakage.The harmful deviation has variance proportional to I(Y_t; R+ | X) and can dominate as training progresses.
  • Background and RLSD: RLSD evaluates its teacher ratio only at the sampled token under stop-gradient, anchoring update signs to the verifier and avoiding vocabulary-wide conditioned sums.Its evidence ratio still uses the student prior as a denominator, creating fluency and one-sided-evidence limitations.
  • Contrastive Evidence Policy Optimization: CEPO replaces RLSD’s single-reference ratio with a contrastive evidence delta comparing correct- and wrong-answer teachers, with the student prior canceling from the differential update.The wrong-answer teacher comes from a rejected rollout already in the batch, so CEPO requires no additional sampling cost.
  • Contrastive Evidence Policy Optimization: Decisive tokens receive larger contrastive credit when the wrong-answer teacher disfavors them, whereas filler tokens remain near unit weight when both teachers track the student prior.The token-level advantage is clipped and inserted into a standard PPO-style surrogate, with the verifier advantage sign preserved.
  • Theoretical guarantees: CEPO preserves direction anchoring and leakage-free gradients, exactly recovers RLSD when the wrong-answer teacher equals the student prior, and adds one teacher pass over RLSD.These properties hold under the stated λ and clipping conditions, while Proposition 1 characterizes strict sharpness over RLSD.

4 Experiments

The experiments evaluate CEPO and four baselines under matched training budgets on Geo3k, using Qwen3-VL models and five held-out multimodal mathematical reasoning benchmarks.

  • Training setup: The study trains on Geo3k, a 3,000-problem geometry dataset with verifiable numeric answers.Training uses LoRA, AdamW, batch size 32, rollout group size G = 8, and maximum sequence length 2,048 tokens.
  • Training setup: CEPO and all baselines use matched LoRA rank, rollout group size, and training steps.CEPO trains Qwen3-VL-2B-Instruct and Qwen3-VL-4B-Instruct for 50 steps on Geo3k, with CEPO-specific hyperparameters λ0 = 0.5, 25-step decay, and ϵw = 0.5.
  • Baselines: The baselines are GRPO, OPSD, SDPO, and RLSD, representing sequence-level reinforcement learning and self-distillation alternatives.OPSD uses KL divergence to a correct-answer teacher, SDPO adds Jensen-Shannon divergence and EMA stabilization, and RLSD is CEPO’s direct predecessor.
  • Evaluation: Evaluation covers DynaMath, LogicVista, MathVisionmini, MMMU, and WeMath as held-out multimodal mathematical reasoning benchmarks.Models are evaluated with lmms-eval using sampling settings including temperature 1.0 and a 32,000-token maximum.

5 Results

CEPO improves average accuracy over GRPO across both model scales and shows its strongest gains on tasks requiring fine-grained multi-step reasoning, while contrastive credit sharpens token-level assignments.

  • Benchmark results: 43.43% average accuracy on Qwen3-VL-2B and 60.56% on Qwen3-VL-4B exceed GRPO’s 41.17% and 57.43%, respectively.The gains are +2.26pp at 2B and +3.13pp at 4B across five benchmarks.
  • Benchmark results: CEPO’s gains are largest on LogicVista at 4B and MathVisionmini at 2B, while MMMU shows the smallest 2B gain of +1.67pp.The paper associates larger gains with fine-grained multi-step reasoning and smaller gains with short reasoning chains.
  • Self-distillation comparison: OPSD and SDPO fall below the untrained 2B baseline, reaching 34.96% and 35.70% versus 39.73%.At 4B, OPSD reaches 56.23% versus the 58.36% base; the paper attributes this pattern to information leakage from vocabulary-wide r-conditioned gradients.
  • Ablations: The actor-policy teacher performs best at 43.43%, compared with 42.18% for a fixed reference and 42.74% for synchronization every 25 steps.These results suggest teacher freshness and on-policy alignment matter more than maintaining a large teacher-student distribution gap.
  • Ablations: Using the verified final answer as r+ and a compact rejected answer as r− yields the strongest feedback-source result at 43.43%, or +2.26pp over GRPO.Full peer rollout negatives reach 42.74%, whereas prefix-only and suffix-only conditioning fall below GRPO at 40.47% and 40.60%.
  • Hyperparameter sensitivity: Performance peaks with evidence clip bounds ϵw ∈ [0.4, 0.5] and degrades toward both extremes.A tight clip collapses CEPO toward GRPO, while large weights destabilize advantage estimation; constant λ = 0.5 and 25-step decay outperform GRPO.
  • Contrastive analysis: Positive contrastive-delta mass rises early while negative-delta mass declines during training.Positive deltas amplify credit on positive-advantage rollouts, whereas negative deltas assign stronger blame on negative-advantage rollouts.
  • Token-level analysis: CEPO suppresses fluent setup prose and concentrates credit on critical algebraic derivations and final-answer tokens.On the geometry trajectory, CEPO’s clip rate is 49.5% versus 71.3% for RLSD, consistent with a wider effective dynamic range.

6 Conclusion

The paper concludes that CEPO provides safe, sharper token-level credit assignment for RLVR and improves performance across five multimodal mathematical reasoning benchmarks.

  • Conclusion: CEPO replaces RLSD’s single-reference evidence ratio with a contrastive ratio between correct- and wrong-answer teachers from rejected training-batch rollouts.The method uses rejected rollouts without additional sampling cost.
  • Conclusion: CEPO preserves RLSD’s direction anchoring and leakage-free gradients while sharpening credit at decisive tokens and leaving filler unchanged.These properties are stated as Theorem 1 and Proposition 1.
  • Conclusion: CEPO outperforms GRPO, OPSD, and SDPO across five multimodal mathematical reasoning benchmarks at 2B and 4B scale.The paper reports that OPSD and SDPO falling below the untrained baseline confirms structural safety as a practical prerequisite.
  • Scope and future work: The reported validation uses Qwen3-VL trained on Geo3k, with larger models, text-only reasoning, and code generation identified as next steps.These extensions are presented as future directions rather than evaluated settings.

A.1 Proof of Theorem 1

The proof establishes that CEPO’s modulated advantage retains the verifier-defined sign and that its sampled-token gradient avoids vocabulary-wide answer-conditioned terms.

  • Direction anchoring: CEPO’s positive weighting and positive clipping bounds preserve the sign of the original advantage unconditionally.Because the modulation factor is positive, sign(Â_t) = sign(A).
  • Leakage-free gradients: The gradient acts only at the sampled token, with no vocabulary-wide sum over r-conditioned weights.This is the structural basis for the method’s leakage-free gradient guarantee.
  • Contrastive evidence: When the teacher and student probabilities match for all tokens, the corresponding contrastive evidence contribution vanishes.The proof explicitly considers the condition P_T(yt) = P_S(yt) for all t.

A.2 Proof of Proposition 1

The proof begins by deriving the relevant condition from Eqs. (7) and (4).

  • The proposition's proof is obtained from Eqs. (7) and (4).
  • The derivation establishes the token-weight comparison used in the proposition.
  • The proof therefore reduces the proposition to an algebraic consequence of the stated equations.

T (yt), wRLSD

CEPO compares teacher support with the policy baseline to sharpen credit or blame. Its advantage is strict when the joint condition holds, but disappears at filler positions.

  • RLSD already assigns above-baseline credit when T(yt) ≥ PS(yt), under the joint condition.
  • CEPO assigns strictly stronger blame to tokens that RLSD already penalizes when T(yt) ≤ PS(yt).
  • Neither method discriminates at informationally neutral filler positions.

B Baseline Hyperparameter Details

The experiments use a shared Qwen3-VL LoRA training infrastructure, with common and method-specific hyperparameters reported separately.

  • All methods use Qwen3-VL-{2B,4B}-Instruct fine-tuned with LoRA through EasyR1.
  • The shared infrastructure uses FSDP and vLLM-accelerated rollout generation.
  • Table 6 reports shared training hyperparameters identical across all five methods.
  • Table 7 reports method-specific hyperparameters, with “—” marking inapplicable parameters.
Loading 2605.19436v1…