PaperlayerView exact revision on arXiv

Source-linked AI summary

When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO

Yu Li, Tian Lan, Zhengling Qi

arXiv:2603.13134v1cs.AI

TL;DR

GRPO overlooks the contrast between correct and incorrect outputs within each sampled group, limiting its use of comparative reasoning signals. The paper reformulates GRPO contrastively, adds BICC for cross-partition conditioning and RCC for covariance-based baseline correction, and reports consistent gains across models, benchmarks, and GRPO variants.

  • Problem

    GRPO evaluates outputs independently relative to the group mean, overlooking contrastive information between correct and incorrect solutions within the same group.

  • Method

    The paper combines BICC, which conditions each sample on opposite-partition outputs, with RCC, which corrects the advantage baseline using reward-confidence covariance.

  • Results

    BICC gains 0.3–1.9 percentage points across settings, while RCC reduces gradient variance by 25–35% across evaluated mathematical reasoning benchmarks.

  • Takeaways & Limitations

    The mechanisms integrate into GRPO variants without additional sampling or auxiliary models and provide consistent improvements across evaluated models and algorithms.

  • Takeaways & Limitations

    The approach is evaluated only on mathematical reasoning with verifiable answers, and its effectiveness depends on reward-signal quality.

Abstract

from arXiv · show

Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models. While it computes advantages based on group mean, GRPO treats each output as an independent sample during the optimization and overlooks a vital structural signal: the natural contrast between correct and incorrect solutions within the same group, thus ignoring the rich, comparative data that could be leveraged by explicitly pitting successful reasoning traces against failed ones. To capitalize on this, we present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples. Building on this insight, we propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces during the optimization, enabling a direct information flow across samples. We further introduce Reward-Confidence Correction (RCC) to stabilize training by dynamically adjusts the advantage baseline in GRPO using reward-confidence covariance derived from the first-order approximation of the variance-minimizing estimator. Both mechanisms require no additional sampling or auxiliary models and can be adapted to all GRPO variants. Experiments on mathematical reasoning benchmarks demonstrate consistent improvements across comprehensive models and algorithms. Code is available at \href{https://github.com/Skylanding/BiCC}{https://github.com/Skylanding/BiCC}.

1. Introduction

GRPO uses grouped relative rewards to train reasoning models but evaluates outputs independently, missing contrastive information between correct and incorrect attempts. The paper introduces BICC for cross-partition conditioning and RCC for covariance-based advantage correction, reporting consistent benchmark gains and reduced gradient variance.

  • GRPO samples multiple candidate solutions and estimates advantages from their relative performance instead of using a separate critic.
  • Vanilla GRPO overlooks contrasting reasoning patterns between correct and incorrect outputs by evaluating each solution mainly against the group mean.
  • BICC reformulates GRPO contrastively and lets correct and incorrect attempts condition each other during training without inference overhead.
  • RCC uses reward-confidence covariance from a variance-minimizing baseline approximation to reduce gradient variance without additional sampling or auxiliary models.
  • 0.3–1.9 percentage points are the consistent BICC gains across evaluated settings, while RCC reduces gradient variance by 25–35%.Experiments use Qwen3-4B and Phi-4-mini across Math500, AMC 2023, AIME 2024, and AIME 2025.

2. Preliminary

The preliminary section defines policy-gradient training for reasoning models and contrasts PPO's learned critic with GRPO's grouped, standardized advantage estimates. It also introduces the baseline principle that motivates later variance correction.

  • The policy πθ generates output sequences autoregressively, while verifiable reasoning tasks use binary rewards indicating correctness.
  • PPO: PPO uses importance sampling and a learned critic to compute token-level advantages relative to state values.
  • GRPO: GRPO removes the critic by sampling G outputs per query and computing one group-standardized advantage for each output.
  • GRPO: The group mean replaces PPO's learned critic as the baseline, while division by σ normalizes advantage scale across queries.
  • A properly chosen baseline can reduce policy-gradient variance without changing the gradient, and the variance-minimizing baseline depends on importance weights and rewards.

3. Another Look at GRPO and Variance Reduction

The paper recasts GRPO as contrastive optimization between correct and incorrect samples, then derives a log-probability approximation for correcting its variance-minimizing baseline. This analysis motivates cross-sample conditioning and covariance-based correction.

  • 3.1. Contrastive Reformulation of GRPO Objective: GRPO groups outputs into correct O+ and incorrect O− partitions according to binary rewards, with group counts determining standardized advantages.
  • 3.1. Contrastive Reformulation of GRPO Objective: The clipped objective applies Cup to positive samples and Clow to negative samples because positive and negative advantages trigger asymmetric clipping behavior.
  • 3.1. Contrastive Reformulation of GRPO Objective: The pairwise reformulation shows that GRPO implicitly performs contrastive optimization over positive-negative pairs, although each ratio still conditions only on the original query.
  • 3.2. Approximating Optimal Baseline under Importance Sampling: The log-probability difference δ replaces direct importance weights because first-order expansion under a trust-region constraint gives w ≈ 1 + δ.
  • 3.2. Approximating Optimal Baseline under Importance Sampling: The corrected baseline is based on reward-log-probability covariance, and positive Cov(R, δ) increases the baseline to prevent high-confidence correct samples from dominating gradients.
  • Figure 1 summarizes standard GRPO, BICC's opposite-partition conditioning, and RCC's covariance-based advantage correction.

4. Method

The method reformulates GRPO as a contrastive objective and enables correct and incorrect solutions to exchange information through bilateral context conditioning. It further applies reward-confidence covariance to correct the advantage baseline and reduce gradient variance.

  • GRPO implicitly optimizes over positive-negative pairs, but evaluates each sample independently without observing other outputs in its group.
  • Bilateral Context Conditioning: BICC constructs augmented contexts by concatenating each query with samples from the opposite reward partition.Correct solutions observe failure traces, while incorrect solutions observe successful traces.
  • Bilateral Context Conditioning: Opposite-partition samples act as privileged training-time information, enabling contrastive learning without inference overhead.The objective retains the standard GRPO structure while conditioning the trainable policy.
  • Bilateral Context Conditioning: The conditioned policy ratio incorporates cross-partition information while preserving deviation measurement from πθold(·|q).When the conditioning weight is 1 for all samples, conditioned GRPO reduces to standard GRPO.
  • Reward-Confidence Correction: RCC adjusts the GRPO baseline using the covariance between reward and confidence, reducing variance when high-confidence correct samples would otherwise dominate gradients.The correction term 2 · Cov(R, δ) is derived from a first-order approximation of the variance-minimizing baseline.
  • Reward-Confidence Correction: RCC omits σ-normalization because covariance provides adaptive scaling, while combining both can over-regularize training.The covariance estimate requires O(G) additional operations per query, and both mechanisms can be applied independently or together across GRPO variants.

5. Result

Experiments evaluate BICC across models, benchmarks, and GRPO variants, finding consistent accuracy gains, stable training, and reduced gradient variance from RCC.

  • Experimental Setup: Experiments use Qwen3-4B and Phi-4-mini across four mathematical reasoning benchmarks, with GRPO variants including Dr.GRPO, DAPO, GMPO, and GSPO.Training uses DAPO-Math-17k, while evaluation covers Math500, AMC 2023, AIME 2024, and AIME 2025.
  • Main Results: BICC yields consistent gains of 0.3–1.9 percentage points across settings, with larger improvements on the weaker Phi-4-mini model.With group size increasing from G = 2 to G = 8, Math500 improvement grows from +0.4% to +0.8% on Qwen3-4B and from +0.6% to +1.9% on Phi-4-mini.
  • Main Results: BICC generalizes across GRPO variants, while BICC-DAPO and BICC-GSPO reach 93.1% and 79.2% on Math500 for Qwen3-4B and Phi-4-mini, respectively.The reported gains remain consistent when conditioning is applied to Dr.GRPO, DAPO, GMPO, or GSPO.
  • Training Dynamics: Conditioned variants show comparable or faster convergence without evident instability despite the longer context, while policy loss decreases and average reward improves during training.Solid lines represent bilateral-conditioning variants and dashed lines represent their baselines.
  • RCC Analysis: RCC adds gains across Pass@k values and reduces gradient variance by 31–36% on Qwen3-4B and 32–37% on Phi-4-mini.A separate analysis reports approximately 25–30% variance reduction and 15–20% faster convergence while maintaining accuracy.
  • RCC Analysis: The separation between correct and incorrect δ distributions widens during training, reaching a larger mean gap for Phi-4-mini than Qwen3-4B.The reported gaps are Δµ = 0.56 for Phi-4-mini and Δµ = 0.27 for Qwen3-4B.

6. Related Work

Related work situates the paper within reinforcement-learning-based reasoning, contrastive preference optimization, GRPO, and policy-gradient variance reduction.

  • Reinforcement Learning for LLM Alignment: Reasoning-oriented LLM training has progressed from RLHF toward reinforcement learning with verifiable rewards and methods used for mathematical reasoning.The related work also discusses chain-of-thought prompting and reasoning-focused systems.
  • Reinforcement Learning for LLM Alignment: Contrastive methods such as DPO and IPO formulate alignment as preference optimization, while GRPO removes PPO’s critic and estimates advantages through group-wise normalization.These approaches frame distinct alternatives for using feedback or relative group performance during optimization.
  • Variance Reduction in Policy Gradients: Variance reduction in policy gradients builds on baselines, whose optimal form generally differs from the simple reward mean, alongside methods such as GAE and RLOO.The literature presents variance reduction as a longstanding concern in policy-gradient optimization.

7. Conclusion

The paper reformulates GRPO contrastively, introduces BICC for cross-partition information flow and RCC for gradient estimation, and reports consistent mathematical-reasoning gains.

  • Contributions: The contrastive reformulation exposes correct and incorrect outputs as interacting subsets within each sampled group.This structure motivates using successful and failed attempts together during training.
  • Contributions: BICC enables cross-partition information flow, while RCC uses reward-confidence correlation to improve gradient estimation.Both mechanisms are designed to integrate with existing GRPO variants with minimal overhead.
  • Results and Scope: Experiments report consistent improvements of 0.3–1.9 percentage points, with larger gains on weaker base models.The conclusion identifies mathematical reasoning benchmarks as the demonstrated evaluation scope.

A. Theoretical Derivations

The derivations recast GRPO as contrastive optimization over correct and incorrect samples, then derive RCC as a covariance-based correction to the variance-minimizing baseline.

  • Binary-reward advantages: Under binary rewards, group-normalized advantages are determined by the proportion of correct samples.For correct samples, A+ = (1 − p̂)/p̂; for incorrect samples, A− = −p̂/(1 − p̂).
  • Contrastive GRPO reformulation: The clipped GRPO objective separates into positive and negative partitions, exposing a contrastive margin between their policy ratios.The pairwise formulation shows optimization over correct–incorrect pairs while each ratio still conditions only on the original query.
  • Variance-minimizing baseline: The variance-minimizing importance-weighted baseline depends on reward-weight correlation rather than only the mean reward.Under the approximation that ψ^2 is independent of reward and importance weight, the baseline is E[Rw^2]/E[w^2].
  • Reward-Confidence Correction: A first-order trust-region expansion yields a baseline correction proportional to Cov(R, δ), where δ is the policy-reference log-probability difference.When covariance is positive, the correction raises the baseline and reduces dominance by high-confidence correct samples; zero covariance recovers standard GRPO.

B.2.2. EVALUATION BENCHMARKS AND PROTOCOL

The evaluation protocol reports Pass@k with confidence intervals and significance tests, while documenting computational costs and cases where BICC falls back to standard GRPO.

  • Evaluation protocol: Pass@k uses 32 generated samples per problem, with 95% confidence intervals estimated by bootstrap resampling.The number of correct samples determines the unbiased Pass@k estimator.
  • Statistical testing: All improvements on Math500 and AMC 2023 are significant at p < 0.001, while AIME results remain significant at p < 0.05.The reported pairwise tests use Wilcoxon signed-rank tests.
  • Computational overhead: BICC increases input sequences by roughly 1.4× at 40% context allocation and proportionally raises memory and computation requirements.The added cost comes from concatenated opposite-partition samples and forward passes over extended contexts.
  • Computational overhead: RCC adds negligible overhead because its O(G) arithmetic uses quantities already available in standard GRPO.The extra subtractions, multiplications, and summations are small relative to forward and backward passes.
  • Fallback behavior: When a group contains only correct or only incorrect outputs, BICC falls back to standard GRPO without additional cost.Such cases are more frequent on very easy or very hard problems.
  • Inference behavior: At inference time, BICC adds zero overhead because the trained model generates from the original prompt alone.Opposite-partition context is used only during training.

C.1. RCC Ablation

The ablation separates BICC’s primary accuracy contribution from RCC’s additional gains and variance reduction, with stronger effects associated with Phi-4-mini’s reward-confidence correlation.

  • Ablation results: RCC adds +0.4% for Qwen and +0.7% for Phi while reducing gradient variance by 31–37%.These results are reported alongside the BICC and RCC ablation on Math500.
  • Reward-confidence correlation: Phi-4-mini reaches Cov(R, δ) = 0.138 at convergence versus 0.066 for Qwen3-4B, matching its larger RCC benefit.The passage explicitly connects stronger reward-confidence correlation with the larger observed RCC improvement.
  • Mechanism: BICC exposes incorrect attempts while evaluating correct ones and correct attempts while evaluating incorrect ones, enabling cross-partition information flow.The conditioning is intended to strengthen systematic reasoning and penalize specific error patterns.
  • Failure modes: BICC benefits most from diverse failure modes, but cannot apply when either partition is empty or when opposite-partition context is heavily truncated.Redundant errors and very long solutions limit the useful conditioning signal.
  • Relation to prior approaches: Unlike self-consistency decoding, BICC uses sample diversity during training and incurs no inference-time context overhead.Its pairwise formulation is also structurally similar to InfoNCE-style objectives while using naturally occurring incorrect attempts.

D.5. Limitations and Future Directions

The paper’s limitations concern binary-reward assumptions, training-cost increases, restricted mathematical evaluation, and dependence on reward quality.

  • Scope and assumptions: The theoretical analysis and experiments assume binary rewards, leaving continuous-reward extensions to soft partition assignments.Using reward magnitude for partition membership is proposed as a natural but more complicated formulation.
  • Efficiency: BICC’s longer input sequences increase training cost, and the overhead may be prohibitive for very large models.The reported accuracy-efficiency trade-off is favorable in the experiments but may not transfer to larger models.
  • Evaluation scope: Evaluation is limited to mathematical reasoning with verifiable answers, so generalization to open-ended tasks with nonbinary correctness remains unexplored.The limitation follows from the paper’s binary-reward setting and benchmark scope.
  • Reward quality: Noisy or misspecified rewards could produce misleading contrastive signals for BICC.The method’s effectiveness therefore depends on reward-signal quality.
  • Future directions: Future directions include selective conditioning weights, cross-query conditioning, and integration with process reward models.These directions target contrastive-value selection, transfer across related problems, and step-level learning.
Loading 2603.13134v1…