PaperlayerView exact revision on arXiv

Source-linked AI summary

CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVR

Sijia Cui, Pengyu Cheng, Jiajun Song, Yongbo Gai, Guojun Zhang, Zhechao Yu, Jianhe Lin, Xiaoxi Jiang, Guanjun Jiang

arXiv:2603.10101v1cs.LGcs.AIcs.CL

TL;DR

RLVR’s binary outcome rewards do not assess intermediate reasoning, creating a gap between final correctness and process quality. CLIPO adds an InfoNCE-based contrastive reward over rollout representations, and experiments report consistent gains across RLVR baselines and diverse reasoning benchmarks.

  • Problem

    RLVR relies on outcome-based rewards without verifying intermediate reasoning steps, allowing process-wrong but outcome-correct rollouts to receive identical positive feedback.

  • Method

    CLIPO adds a lightweight contrastive head and InfoNCE objective that aligns successful reasoning trajectories, separates erroneous ones, and supplies a dense auxiliary reward.

  • Results

    CLIPO consistently improves RLVR baselines across diverse mathematical reasoning benchmarks, especially under challenging and distribution-shifted conditions.

  • Takeaways & Limitations

    Contrastive rewards provide complementary, fine-grained trajectory signals that generalize across underlying RLVR algorithms and support more robust reasoning optimization.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capacity of Large Language Models (LLMs). However, RLVR solely relies on final answers as outcome rewards, neglecting the correctness of intermediate reasoning steps. Training on these process-wrong but outcome-correct rollouts can lead to hallucination and answer-copying, severely undermining the model's generalization and robustness. To address this, we incorporate a Contrastive Learning mechanism into the Policy Optimization (CLIPO) to generalize the RLVR process. By optimizing a contrastive loss over successful rollouts, CLIPO steers the LLM to capture the invariant structure shared across correct reasoning paths. This provides a more robust cross-trajectory regularization than the original single-path supervision in RLVR, effectively mitigating step-level reasoning inconsistencies and suppressing hallucinatory artifacts. In experiments, CLIPO consistently improves multiple RLVR baselines across diverse reasoning benchmarks, demonstrating uniform improvements in generalization and robustness for policy optimization of LLMs. Our code and training recipes are available at https://github.com/Qwen-Applications/CLIPO.

1. Introduction

RLVR rewards final outcomes but does not verify intermediate reasoning, motivating CLIPO’s contrastive alignment of successful trajectories to capture shared reasoning structure. The framework supplements outcome feedback with trajectory-level consistency signals and is evaluated across diverse reasoning benchmarks.

  • Motivation: RLVR uses outcome-based rewards without verifying intermediate reasoning steps, leaving logically sound and spurious paths indistinguishable when they share an outcome.This coarse binary signal has raised concerns about generalization and answer memorization.
  • Motivation: Contrastive learning motivates treating successful and failed trajectories as positive and negative instances to enhance RLVR generalization.The approach maximizes similarity among positives while separating negatives.
  • CLIPO: CLIPO integrates contrastive learning into group-based policy optimization through a lightweight auxiliary head and an InfoNCE objective.The objective aligns correct trajectories and enforces a margin from erroneous ones.
  • CLIPO: CLIPO converts the contrastive loss into a dense auxiliary reward that complements sparse outcome feedback and provides a more informative policy-optimization gradient.The method distills shared logical structure across diverse successful rollouts without requiring process annotations or additional supervision.
  • Evaluation: The framework is evaluated in two tracks: GSM8K training with evaluation on eight benchmarks, and MATH 7.5K training with broader reasoning evaluation.The supplied passage introduces both experimental tracks but truncates the second track’s evaluation description.

2. Preliminary

The preliminaries describe RLVR’s binary verifier feedback and policy-gradient optimization, then introduce contrastive learning as representation learning that clusters similar samples and separates dissimilar ones. CLIPO combines these ideas by computing trajectory-level contrastive rewards from rollout embeddings.

  • Reinforcement Learning with Verifiable Rewards: RLVR directly uses deterministic environment feedback, such as unit tests or equivalence checkers, to produce binary reward outcomes.The reward checks whether the predicted answer is equivalent to the ground-truth answer.
  • Policy Optimization: Policy-gradient methods such as PPO and GRPO optimize RLVR objectives with REINFORCE-style estimators based on rollout rewards and token advantages.GRPO estimates a baseline from a group of sampled responses and assigns the normalized group-based advantage across rollout tokens.
  • Policy Optimization: GSPO shifts optimization from token-level importance weighting to the sequence level to mitigate variance in large-scale training.It is presented as a stability and scalability improvement to GRPO.
  • Reinforcement Learning with Verifiable Rewards: RLVR’s sparse binary feedback lacks the resolution to distinguish the quality of intermediate reasoning steps.This limitation motivates augmenting outcome rewards with finer-grained trajectory signals.
  • Contrastive Learning: Contrastive learning maps semantically similar samples close together and dissimilar samples apart in a latent representation space.InfoNCE frames this objective as identifying a positive sample among contrastive negatives through multiclass classification.
  • CLIPO Framework: CLIPO applies a contrastive head to rollout hidden states, produces trajectory-level embeddings, and computes contrastive rewards from similarities among successful and failed trajectories.These rewards are combined with the original RLVR rewards in the final CLIPO reward.

3. Methodology

CLIPO augments group-based policy optimization with contrastive learning over rollout trajectories. It aligns successful reasoning paths, separates erroneous ones, and adds a dense auxiliary reward to guide more consistent reasoning.

  • CLIPO Objective: CLIPO applies an InfoNCE objective within each rollout group to maximize consistency among correct trajectories and enforce separation from erroneous ones.Successful rollouts form positives, while other group rollouts serve as negatives.
  • CLIPO Objective: The method estimates mutual information among successful rollouts using group-based sampling because the exact joint distribution is intractable.For each positive rollout, another successful rollout is sampled as its positive pair, and remaining group rollouts provide negatives.
  • CLIPO Implementation: CLIPO applies its contrastive loss only when a rollout group contains more than one successful response and at least one failure.This avoids cases without positive pairs or without negatives.
  • CLIPO Implementation: A lightweight contrastive head transforms pooled transformer hidden states into semantic embeddings used to calculate trajectory similarity.Mean pooling produces a response-level representation before projection into the contrastive embedding space.
  • CLIPO Implementation: The contrastive reward is added to the original verifiable reward, with reweighting and lower-threshold clipping to prevent the auxiliary signal from dominating.The resulting reward provides granular guidance among multiple correct responses and favors trajectories within a consensus region.

4. Experiments

CLIPO is evaluated across two reasoning tracks, multiple RLVR baselines, model architectures, contrastive-head settings, and temperature values. It consistently improves challenging and distribution-shifted reasoning performance, while ablations show that jointly trained heads and suitable temperatures matter.

  • Experimental Setup: The experiments use two tracks: GSM8K and general reasoning, plus competition-level mathematical reasoning with varying difficulty.Track I evaluates distribution-shifted GSM8K variants and general benchmarks; Track II evaluates competition-level mathematics.
  • Experimental Setup: CLIPO is compared with GRPO, GSPO, DAPO, and other group-based RLVR baselines using Pass@1 evaluation.GRPO is the default base method, and Δ denotes the relative improvement from adding CLIPO.
  • Main Results: 63.26 is the highest overall average score achieved by GRPO+CLIPO on GSM8K and general reasoning, with gains of +1.48 and +3.36 on GSM8K-P1 and GSM8K-P2.GRPO+CLIPO also achieves the highest math-average score of 75.35 and best or second-best performance across all four general reasoning benchmarks.
  • Main Results: 44.05 is the highest competition-level average score achieved by DAPO+CLIPO, while CLIPO improves overall averages for GRPO, GSPO, DAPO, and GMPO by +1.35, +0.80, +1.20, and +0.83.CLIPO also reports average improvements across competition-level datasets for the four RLVR methods.
  • Main Results: CLIPO gains are consistent across tracks, particularly on perturbation, symbolic, and other distribution-shifted mathematical benchmarks.The reported results associate contrastive rewards with clustering high-quality solutions and separating low-quality ones in representation space.
  • Ablation Studies and Analysis: Freezing the contrastive head reduces Track I averages by −0.62, −0.92, and −0.77, and Track II averages by −1.03, −0.91, and −0.97.The ablation supports jointly optimizing the contrastive head with the base model.
  • Ablation Studies and Analysis: CLIPO improves average scores by +0.53 for DS-7B and +1.31 for Llama-8B, while DS-7B’s Math-Avg. decreases by −0.11.The authors speculate that extensive prior mathematical fine-tuning and equal training duration may contribute to overfitting on some DS-7B tasks.

5. Conclusions

CLIPO extends RLVR with contrastive learning to move beyond coarse outcome-based supervision. Experiments across mathematical benchmarks show consistent gains in robustness and generalization, including on symbolic, perturbed, and out-of-distribution tasks.

  • 5. Conclusions: CLIPO uses a lightweight contrastive head with an InfoNCE objective to align successful reasoning trajectories and separate erroneous ones.The resulting dense contrastive reward complements standard verifier feedback.
  • 5. Conclusions: CLIPO reshapes RLVR feedback from sparse and binary to informative and relational without requiring external process annotations or additional supervision.
  • 5. Conclusions: Consistent gains on symbolic, perturbed, and out-of-distribution tasks support more robust and generalizable reasoning with contrastive reward shaping.

A.1.1. Baseline Implementations

The experiments reproduce several policy-optimization baselines under shared training settings, while preserving each method’s distinct objective and aggregation scheme. The appendix also specifies default configurations and contrastive-head parameters.

  • A.1.1. Baseline Implementations: GRPO, GSPO, DAPO, and GMPO are reproduced using the public verl framework.Baselines share data, initialization, sampling, batch, optimization, and distributed-training settings.
  • A.1.1. Baseline Implementations: DAPO uses asymmetric clipping with 𝜖𝑙= 0.2 and 𝜖ℎ= 0.28, while disabling KL reward shaping and KL loss regularization.
  • A.1.1. Baseline Implementations: GMPO uses a geometric-mean policy objective, symmetric clipping with 𝜖= 0.4, and token-level advantage aggregation.
  • A.1.1. Baseline Implementations: A unified protocol fixes learning rate, rollout sampling, batch size, sequence length, and distributed memory optimization across baselines.This isolates differences to policy objectives and advantage aggregation methods.

A.1.2. Prompt Construction

All examples are converted into a unified instruction-following format with dataset-specific prompts. Mathematical tasks request step-by-step reasoning and boxed answers, while multiple-choice tasks specify the permitted answer format.

  • A.1.2. Prompt Construction: Each example combines the original problem with a dataset-specific instruction in a user-only prompt.No system or assistant messages are used.
  • A.1.2. Prompt Construction: Most mathematical datasets instruct models to think step by step and place the final answer within \boxed{} delimiters.
  • A.1.2. Prompt Construction: Multiple-choice prompts additionally specify answer types, such as A, B, C, or D for MMLU.

A.1.3. Verifiable Reward Models

The verifiable reward model extracts the LLM’s final answer and compares it with ground truth. It assigns a binary reward based on correctness.

  • A.1.3. Verifiable Reward Models: The reward model extracts the final answer enclosed within \boxed{} delimiters.
  • A.1.3. Verifiable Reward Models: The extracted prediction, LLMpred, is compared against the ground-truth answer GT.
  • A.1.3. Verifiable Reward Models: The reward r equals 0 for incorrect answers and 1 for correct answers.

A.2. Contrastive Head Representation and Ablation

The contrastive head learns a structured embedding space in which correct reasoning trajectories cluster and incorrect ones separate. Ablations and sensitivity studies show that the learned mapping, temperature, group size, and model backbone affect performance and stability.

  • Representation analysis: After contrastive training, correct rollout embeddings cluster closely while incorrect embeddings become relatively separated.Before training, correct and incorrect trajectories are mixed without clear structure; afterward, the embeddings show coherent clustering.
  • Representation analysis: The learned semantic manifold can encode trajectory quality and support more informative credit assignment for reasoning paths sharing intermediate structure.This provides an inductive structure beyond binary outcome feedback.
  • Fixed-head ablation: Freezing the contrastive head causes performance degradation across average metrics, including decreases of -1.12 and -2.48 on P1 and P2.Additional decreases of -1.03, -0.91, and -0.97 occur on M-Avg, C-Avg, and Avg, respectively.
  • Contrastive loss variants: SoftNN and SupCon are evaluated as alternative contrastive losses, while SoftNN shows limited improvements on G-Avg. tasks and some decreases.The SoftNN ablation investigates this behavior across different RLVR base methods.
  • Sensitivity analysis: Higher temperature values produce greater fluctuations in positive-pair similarity, whereas larger group sizes consistently improve performance across temperature settings.The temperature results indicate increased instability in distinguishing positive and negative examples.
  • Base-model evaluation: CLIPO is also validated on DS-7B and Llama-8B in addition to Qwen2.5.The cross-backbone comparison is reported in Table 19.
Loading 2603.10101v1…