Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective

Equivalent discriminative view

• GRPO can be rewritten as weighted positive-negative discrimination, with binary-reward weighting scaled by p(q)(1 − p(q)).The associated scores are averages of clipped token-level importance ratios.Source: §1 · p. 1, Figure 1 · p. 2

Objective limitations

• GRPO optimizes clipped ratio-based surrogate scores rather than sequence likelihoods, allowing surrogate separation without improving likelihood ordering under the current policy.This creates a training-inference mismatch that weakens the connection between objective improvement and generation quality.Source: §1 · p. 1
• GRPO assigns identical coefficients to all positive rollouts and another shared coefficient to all negatives, ignoring within-group score gaps.The coefficients are tied to the group pass rate rather than whether a positive is close to negatives or a negative is especially high-scoring.Source: §1 · pp. 1–2, Figure 2 · p. 3

ConSPO

• ConSPO replaces clipped ratio scores with length-normalized sequence log-probabilities and contrasts verified positives with same-group negative distractors using an InfoNCE-style objective.Its softmax weighting strengthens updates for poorly separated positives and higher-scoring incorrect rollouts.Source: §1 · p. 2
• ConSPO schedules a margin from zero to preserve separation pressure after positives and negatives become correctly ordered but remain insufficiently separated.The curriculum shifts optimization from coarse ordering toward stronger separation.Source: §1 · p. 2

Experiments

• ConSPO consistently outperforms strong RLVR baselines across models, scales, and training datasets on challenging reasoning benchmarks.The training dynamics in Figure 2 also show consistently higher rewards than GRPO during training.Source: §1 · p. 2, Figure 2 · p. 3

RLVR for LLM reasoning

• Prior work modifies GRPO through loss reallocation, importance-sampling mechanisms, and reward shaping for more stable or efficient reasoning.These approaches primarily adjust the existing GRPO framework.Source: §2 · p. 3
• DisCO reinterprets GRPO discriminatively and explores alternative scoring functions but does not address score-insensitive credit assignment.ConSPO targets that limitation with an InfoNCE-style objective, likelihood-aligned scores, and a curriculum margin.Source: §2 · p. 3

Contrastive objectives for policy alignment

• ConSPO applies contrastive optimization to online RLVR, constructing positive-negative comparisons from verifiable rewards on policy-generated rollouts.This differs from offline preference optimization methods that use static pairwise preference data.Source: §2 · p. 3

GRPO setup

• Rollouts are generated autoregressively from πold(· | q) and receive verifiable sequence-level rewards r_i = r(o_i, q).Each rollout is represented as a sequence of tokens.Source: §3 · p. 3
• The group-relative advantage is computed by standardizing each rollout reward using the sampled group mean and variance.The supplied formulation gives Â_i = (r_i − μ̂_q)/σ̂_q.Source: §3 · p. 3

Expected analysis

• The expected reward distribution under the old policy is characterized by μ_q and σ_q^2, its mean and variance over rollouts.These expected quantities support the theoretical analysis of GRPO.Source: §3 · p. 3

Policy update

• Policy parameters are updated by maximizing a clipped surrogate objective with a KL penalty relative to a reference policy.The clipped surrogate is later analyzed in expectation form after removing the KL regularizer.Source: §3 · p. 3

4.1 An Equivalent Discriminative Reformulation of GRPO

• GRPO admits an equivalent weighted positive-negative discrimination formulation for valid queries, including a binary-reward specialization.The reformulation uses positive and negative reweightings of the old policy and sequence-level surrogate scores formed from clipped token-level importance ratios.Source: §4.1 · p. 4
• The reformulation identifies likelihood-misaligned surrogate scores because GRPO separates rollouts using clipped importance-ratio averages rather than sequence likelihoods governing generation.Increasing the surrogate score gap therefore does not necessarily improve the current policy’s likelihood ordering of positive and negative rollouts.Source: §4.1 · p. 4
• GRPO’s credit assignment is insensitive to within-group score gaps: all positives share one coefficient and all negatives another, regardless of hard negatives or poorly separated positives.The coefficients are tied to empirical pass-rate statistics rather than current positive-negative score gaps.Source: §4.1 · pp. 4–5

4.2 A Contrastive View of RLVR Optimization

• ConSPO replaces clipped-ratio scores with length-normalized sequence log-probabilities and contrasts verified positives against negative distractors using a group-wise InfoNCE-style objective.The shared likelihood scale matches the sequence probabilities governing autoregressive generation, while softmax normalization compares each positive with the negative set.Source: §4.2 · p. 5
• InfoNCE credit adapts to current score comparisons: updates strengthen for poorly separated positives and high-scoring negatives, unlike GRPO’s fixed reward-statistic coefficients.Positive gradients attenuate as separation improves, while negative suppression grows when positives are poorly separated and is larger for higher-scoring negatives.Source: §4.2 · p. 5

4.3 Scheduled Margin as a Separation Curriculum

• ConSPO schedules a nonnegative margin from zero to a target M during the first α portion of training, then holds it fixed to preserve separation pressure.Subtracting the margin from positive scores delays gradient attenuation until positives exceed distractors by a larger margin.Source: §4.3 · pp. 5–6

5.1 Settings

• The experiments cover five model families or scales, seven reasoning benchmarks, two training datasets, and seven recent RLVR baselines.AIME, HMMT, and AMC use avg@32, while the remaining benchmarks use pass@1; all-positive and all-negative groups are discarded during training.Source: §5.1 · pp. 6–7

5.2 Main Results

• 44.4 average performance on DeepSeek-R1-Distill-Qwen-1.5B exceeds GRPO by 4.4 points and the strongest baseline by 1.6 points, with ConSPO best on all seven benchmarks.It also improves over the strongest baseline by 2.1 points on the 7B model and 1.9 points on the 8B model.Source: §5.2 · p. 8
• 34.0 average performance on Qwen3-4B-Base exceeds the strongest baseline by 2.1 points, while HMMT25 rises from 0.9 to 8.0.ConSPO also reaches 29.2 on Llama-3.2-3B-Instruct, improves by 0.6 points over the strongest baseline, and achieves 44.1 on DAPO-Math-17k, 1.9 points higher.Source: §5.2 · p. 8

5.3 Ablation Study

• 42.7 average performance without the contrastive objective and 43.4 without likelihood-aligned scores, versus 44.4 for full ConSPO.Removing the margin gives 44.1, while a fixed margin gives 43.9, indicating complementary contributions from all three components.Source: §5.3 · pp. 8–9, Table 6 · p. 8

5.4 Parameter Study

• Too-small contrastive temperature can destabilize optimization, while large target margins or warmup ratios reduce performance.The study therefore supports moderate separation requirements introduced at an appropriate pace.Source: Figure 3 · p. 8, §5.4 · p. 9

• ConSPO addresses GRPO’s objective-level limitations with likelihood-aligned sequence scores, a group-wise InfoNCE-style objective, and a curriculum-scheduled margin.Source: §6 · p. 9

Limitations

• Validation primarily focuses on models up to 8B, while systematic evaluations at 32B and beyond remain future work.The paper additionally compares ConSPO and GRPO on a 32B model in Appendix D.Source: Limitations · p. 9
• ConSPO is a preliminary contrastive-RLVR exploration using one InfoNCE-style objective, likelihood-aligned scores, and a curriculum-scheduled margin.Broader objective formulations and rollout scoring functions remain worth investigating.Source: Limitations · p. 9

Ethical Considerations

• The study introduces no datasets containing private, sensitive, or personally identifiable information.Source: Ethical Considerations · p. 9
• Potential harms resemble those of general LLM reasoning systems, including misuse or insufficiently supervised deployment.The authors emphasize careful evaluation and safeguards for fair, safe, and responsible use.Source: Ethical Considerations · p. 9

A.1 Proof of Proposition 1 and Lemma 1

• The Lemma 1 proof establishes equal positive and negative reweighting totals and positivity under nonzero advantage variance.Source: §A.1 · p. 12
• The appendix derives the GRPO reformulation by removing KL regularization, applying the clipping identity, and specializing to binary rewards.The binary setting uses verified positive and negative outcomes with 0 < p(q) < 1.Source: §A.1 · p. 13

A.2 Proof of Lemma 2

• Lemma 2 differentiates the group-wise contrastive probabilities with respect to positive and negative rollout scores to obtain the stated gradient identities.Source: §A.2 · p. 14

B Detailed Related Work

• Unlike DisCO, ConSPO addresses score-insensitive credit assignment with an InfoNCE-style online RLVR objective, likelihood-aligned scores, and a curriculum margin.Source: B Detailed Related Work · p. 15
• ConSPO transfers contrastive learning from offline preference pairs to positive-negative comparisons formed from verifiable rewards on policy-generated rollouts.Source: B Detailed Related Work · p. 15

C.1 Benchmarks

• The evaluation measures mathematical reasoning on seven benchmarks, including AIME24–26, HMMT25, MATH500, and OlympiadBench, covering competition mathematics and advanced scientific reasoning.AIME24–26 contains challenging high-school competition problems, while OlympiadBench contains challenging mathematics and physics problems and evaluates advanced scientific reasoning.Source: §C.1 · pp. 15–16
• Additional 32B results report avg@32 on AIME, HMMT, and AMC and pass@1 on the remaining benchmarks.The table caption abbreviates OlympiadBench as O-Bench.Source: Table 9 · p. 16
• AIME, HMMT, and AMC use avg@32, while the remaining benchmarks use pass@1 to report performance.The avg@32 choice reduces variance on relatively small test sets.Source: §C.1 · p. 16

C.2 Implementation Details

• R1-Distill, Qwen3-4B-Base, and Llama-3.2-3B-Instruct use shared configurations with a 4096-token maximum response length for the latter two models.Qwen3-4B-Base is trained for 2 epochs, whereas the other models use 4 epochs.Source: §C.2 · p. 16

D Additional Experimental Results

• The additional experiment evaluates scalability using DeepSeek-R1-Distill-Qwen-32B with the same data and hyperparameter settings as the main experiments.GRPO and ConSPO are each trained for two epochs.Source: D Additional Experimental Results · p. 16
• ConSPO improves the average score from 60.2 to 63.0 over GRPO across all seven benchmarks on DeepSeek-R1-Distill-Qwen-32B.Both methods use the same training data and hyperparameter settings and are trained for two epochs.Source: D Additional Experimental Results · p. 16

E Prompt Template

• A unified prompt template is used across methods for both rollout generation and evaluation.The prompt asks models to solve step by step and place the final answer within \boxed{}, facilitating answer extraction and rule-based verification.Source: E Prompt Template · p. 16, Figure 4 · p. 16