PaperlayerView exact revision on arXiv

Source-linked AI summary

It Takes Two: Your GRPO Is Secretly DPO

Yihong Wu, Liheng Ma, Lei Ding, Muzhi Li, Xinyu Wang, Kejia Chen, Zhan Su, Zhanguang Zhang, Chenyang Huang, Yingxue Zhang, Mark Coates, Jian-Yun Nie

arXiv:2510.00977v3cs.LGcs.CL

TL;DR

The paper examines whether GRPO’s effectiveness depends on large groups for accurate advantage estimates or instead on an implicit contrastive objective. The paper theoretically analyzes GRPO as contrastive gradient optimization and introduces 2-GRPO, which uses two rollouts per prompt. 2-GRPO achieves performance comparable to 16-GRPO while substantially reducing training time.

  • Problem

    The paper examines whether GRPO’s effectiveness depends on large groups for accurate advantage estimates or instead on an implicit contrastive objective.

  • Method

    The paper theoretically analyzes GRPO as contrastive gradient optimization and introduces 2-GRPO, which uses two rollouts per prompt.

  • Results

    2-GRPO achieves performance comparable to 16-GRPO while substantially reducing training time.

  • Takeaways & Limitations

    The findings support contrastive formulation as a more principled account of GRPO and indicate that large group sizes are not essential.

  • Takeaways & Limitations

    2-GRPO can discard groups with matching rewards, wasting samples when the policy is highly accurate; resampling is introduced to address this.

Abstract

from arXiv · show

GRPO has emerged as a prominent reinforcement learning algorithm for post-training LLMs. Unlike critic-based methods, GRPO computes advantages by estimating the \emph{value baselines} from group-level statistics, eliminating the need for a critic network. Consequently, the prevailing view emphasizes the necessity of large group sizes, which are assumed to yield more accurate statistical estimates. In this paper, we propose a different view that the efficacy of GRPO stems from its implicit contrastive objective in the optimization, which helps reduce variance via the control variate method. This makes GRPO structurally related to preference learning methods such as DPO. This perspective motivates 2-GRPO, a minimal group-size variant that constructs contrastive signals with only two rollouts. We provide a rigorous theoretical analysis of 2-GRPO and empirically validate its effectiveness: 2-GRPO retains $97.6\%$ of the performance of 16-GRPO, while requiring only $12.5\%$ of the rollouts and $21\%$ of the training time.

1 Introduction

The paper argues that GRPO works primarily through an implicit contrastive objective, not accurate group-level value-baseline estimation. This view motivates 2-GRPO, which uses two rollouts and achieves comparable performance with substantially lower resource use.

  • GRPO removes the critic network by estimating advantages from normalized reward statistics across responses sampled for one prompt.This design avoids the memory and computational overhead of a value network while retaining strong reasoning-task performance.
  • GRPO’s efficacy primarily arises from its implicit contrastive objective, which reduces gradient-estimate variance through a control-variate mechanism.The group mainly constructs contrastive sample pairs rather than accurately estimating value baselines.
  • 2-GRPO uses the minimal group size of two to construct contrastive signals aligned with DPO.The paper theoretically analyzes 2-GRPO and empirically evaluates it across models and tasks.
  • 2-GRPO achieves performance comparable to 16-GRPO while substantially reducing rollouts and training time.The introduction also presents 2-GRPO+RS, which improves performance by reducing discarded samples while remaining more efficient than 16-GRPO.

2 Preliminary

The paper studies RL post-training for reasoning LLMs, where trajectory-level rewards guide policy optimization. It contrasts high-variance vanilla policy gradients and critic-based PPO with critic-free GRPO, which estimates advantages from group reward statistics.

  • 2.1 Problem Setting and Notation: The setting maximizes expected trajectory-level rewards for LLM responses, focusing mainly on verifiable rewards that mark answers correct or incorrect.A trajectory concatenates a prompt with its generated response.
  • 2.2 The Story of Variance Reduction: VPG, PPO, and GRPO: Vanilla policy gradients can have high-variance estimates and unstable training, motivating advantage-based variance reduction.The advantage subtracts a value baseline from each reward.
  • 2.2 The Story of Variance Reduction: VPG, PPO, and GRPO: PPO uses an auxiliary critic network to estimate value baselines, while importance sampling and clipping support near-on-policy optimization.The clipping function constrains the importance-sampling ratio within [1 − ε, 1 + ε].
  • 2.2 The Story of Variance Reduction: VPG, PPO, and GRPO: GRPO estimates advantages from the mean and standard deviation of rewards among grouped responses, eliminating the critic network’s computational and memory costs.This commonly motivates using sufficiently large groups for more accurate group-level statistics.

3 A Tale of Two Algorithms: GRPO and DPO

The paper shows that GRPO and DPO implement the same contrastive policy-gradient principle under online and offline RL settings. Their apparent differences reflect adaptations in sampling, weighting, aggregation, and regularization rather than distinct core objectives.

  • DPO is a 1-vs-1 contrastive method based on offline human-annotated preference pairs.Its objective uses a sigmoid weighting over the preferred and unpreferred responses.
  • GRPO supports dynamic N-vs-M sampling because online groups may contain arbitrary numbers of positive and negative responses.The group size determines the Monte Carlo sample count for estimating the same positive and negative gradients.
  • 3.2 GRPO: N-vs-M Contrastive Learning: GRPO’s objective is equivalent to minimizing an N-vs-M contrastive-loss estimator under binary rewards.The formulation treats correct and incorrect trajectories as positive and negative samples with dynamic counts.
  • 3.2 GRPO: N-vs-M Contrastive Learning: The contrastive interpretation extends beyond binary rewards to continuous rewards, although the paper focuses its analysis on verifiable-reward settings.The binary-reward assumption is used to align the derivation with RLVR.
  • GRPO and DPO both increase preferred-output likelihood relative to unpreferred outputs while using different Monte Carlo estimators.GRPO operates online with generated rollouts, whereas DPO uses offline, pre-collected preference pairs.
  • GRPO and DPO differ in implementation details including token aggregation, importance-sampling correction, group weighting, and reference-model regularization.These differences are presented as choices associated with online versus offline learning regimes.

4 Why Viewing GRPO From Contrastive Learning?

The paper explains GRPO’s variance reduction through contrastive control variates and tests whether large groups are truly necessary. Two-rollout GRPO remains effective and substantially more efficient, though reward agreement can cause sample waste.

  • GRPO’s contrastive gradient acts as a control variate, reducing estimator variance when its coefficient lies in an appropriate range.The resulting variance reduction stabilizes RL training.
  • Variance reduction depends on correlation between positive and negative gradients, which are generated by the same model for the same prompt.This shared conditioning typically induces nontrivial correlation.
  • 4.2 GRPO with Small Group Size: It Should Fail, But Doesn’t: The contrastive view predicts that small groups can remain effective because Monte Carlo gradient estimates are unbiased regardless of sample size.This contrasts with the value-baseline view, which predicts failure from unreliable small-group statistics.
  • 4.3 Introducing 2-GRPO: With two rollouts, opposing rewards produce advantages +1 and −1, while matching rewards produce zero advantages, yielding an online DPO counterpart.The resulting signal is binary and contrastive.
  • 4.3 Introducing 2-GRPO: 2-GRPO discards groups whose two rewards match, wasting samples when the policy is highly accurate; 2-GRPO+RS replaces discarded groups with fresh prompts.Resampling adds rollout computation but modestly lifts peak performance, while optimization-stage cost remains that of 2-GRPO.
  • 4.3 Introducing 2-GRPO: 2-GRPO reaches 97.6% of 16-GRPO’s average performance using 12.5% of its rollouts and 21.0% of its training time.Table 1 covers models post-trained on MATH or DAPO-Math-Sub and evaluated on five math reasoning benchmarks after 10 epochs.

5 Experiments

Experiments evaluate 2-GRPO across mathematical, vision-reasoning, and code-generation tasks, finding comparable performance to 16-GRPO with substantially lower rollout and training costs.

  • 5.1 Math Reasoning: 97.6% of 16-GRPO’s average performance is achieved using 12.5% of its rollouts and 21.0% of its training time.Table 1 reports Mean@32 and training time for the main comparison.
  • 5.1 Math Reasoning: 2-GRPO+RS outperforms 16-GRPO on average while using roughly half its training time.Resampling improves peak performance but is slower than standard 2-GRPO.
  • 5.1 Math Reasoning: 2-GRPO’s Pass@K performance is comparable to 16-GRPO across five math benchmarks and different K choices.It outperforms 16-GRPO on AMC 2023 and Olympiad Bench, while AIME 2025 results depend on the training dataset.
  • 5.1 Math Reasoning: 2-GRPO remains effective and efficient on Geometry3K and Code-R1, converging substantially faster than 16-GRPO.The reduced samples generated and updated per step accelerate learning beyond math reasoning.
  • 5.2 Ablation Study: The Effect of Group Size: Across group sizes G = 2, 4, 8, 16, Mean@32 differences remain consistently small, and larger groups do not reliably improve Pass@32.The ablation uses a fixed configuration, though smaller groups receive fewer rollouts per training mini-batch and therefore higher gradient-estimate variance.

6 Conclusion

The paper concludes that GRPO is fundamentally a contrastive optimization method, linking it structurally to DPO and motivating the efficient two-rollout 2-GRPO variant. Its conclusions concern RL-based LLM post-training, while small-sample normalization and clipping remain important technical considerations.

  • 6 Conclusion: GRPO’s group mechanism primarily constructs contrastive signals rather than accurately estimating value baselines, connecting GRPO and DPO through a shared gradient principle.GRPO corresponds to online RL and DPO to offline RL under this interpretation.
  • 6 Conclusion: 2-GRPO uses two rollouts per prompt and remains theoretically motivated despite being degenerate under traditional advantage estimation.The paper reports comparable performance to 16-GRPO with lower rollout-generation and policy-optimization overhead.
  • 6 Conclusion: With n = 2, normalization can have infinite variance and no defined mean because its variance factor follows a Cauchy distribution.This small-sample instability undermines statistical stability even though the paper’s contrastive analysis motivates 2-GRPO.
  • 6 Conclusion: The GRPO gradient is characterized as a Monte Carlo estimator of a contrastive derivative.The derivation rewrites the objective by separating positive and negative trajectories, whose advantages are constant within each group.
  • 6 Conclusion: Importance sampling adjusts log-likelihood terms for online near-on-policy learning, while DPO directly uses log-likelihood in its offline setting.Clipping is an additional modification applied for training stability.

B.5 Proof of Lemma B.1: DPO is 1-vs-1 contrastive learning

The proof frames DPO as a 1-vs-1 contrastive loss and relates its gradient structure to GRPO, while distinguishing their online and offline coefficient choices.

  • DPO is a 1-vs-1 contrastive loss estimator.
  • GRPO and DPO share an underlying contrastive optimization mechanism despite differing coefficient instantiations.
  • Online GRPO uses importance sampling and explicit KL regularization, whereas offline DPO uses log-likelihood terms and incorporates reference regularization implicitly into its advantage.
  • The analysis treats 2-GRPO as valid under contrastive learning even though its value-estimate interpretation is degenerate.
  • 2-GRPO’s advantage estimates differ from standard GRPO by a scaling factor whose benefit remains an open question.

C.2 Key of Variance Reduction: the Training Batch Size, not the Group Size

The variance analysis argues that optimization mini-batch size, measured in rollouts, matters more than per-prompt group size. Smaller groups can preserve the effective rollout batch and total sampling budget.

  • The effective optimization sample count is the number of rollouts in the mini-batch, not merely the number of prompts.
  • A larger rollout-count mini-batch B lowers the variance of the empirical gradient estimate.The comparison states that if B1 < B2, then Var[ĝ_B1] > Var[ĝ_B2].
  • When group size G decreases, increasing the number of prompts Q can preserve B = QG without increasing total per-epoch computational cost.
  • Under a fixed total-rollout budget, 2-GRPO is not less likely than 16-GRPO to sample a correct answer across training.
  • For difficult questions, more frequent policy updates may preserve effectiveness and adapt updates to varying inputs under 2-GRPO.

D.1 Experiment Details

The experiments evaluate RL post-training across mathematical, visual reasoning, and code generation tasks using specified models, datasets, benchmarks, and computational-cost proxies.

  • Math experiments use Qwen2.5-Math-1.5B and 7B models trained on MATH and DAPO-Math-17k, then evaluated on five reasoning benchmarks.
  • Visual reasoning uses Qwen2.5-7B with Geometric3K, while code generation uses Qwen2.5-7B-Instruct-1M with code-r1-12k7.
  • Training and evaluation use specified sampling, sequence-length, optimizer, learning-rate, and GRPO configurations within the verl framework.
  • The study compares computational cost using the total number of training rollouts.For a fixed model and GRPO algorithm, rollout count is treated as proportional to FLOPs.

D.3 Sample Discard Rate of 2-GRPO

The discard-rate analysis examines when 2-GRPO fails to form useful contrastive pairs, using a Qwen-7B model post-trained on MATH as a representative case.

  • 2-GRPO may have a high discard rate when prompts are extremely easy or extremely difficult for the model.
  • The average discard rate in the RLVR setting can be estimated from the average reward.
  • The study reports 2-GRPO’s discard rate across training steps for Qwen-7B post-trained on MATH.

D.4 Ablation Study on Group Size

The ablation study examines how group size affects training rewards and validation performance on MATH. Figure 5 tracks both quantities throughout training for different group sizes.

  • D.4 Ablation Study on Group Size: Figure 5 compares reward curves and validation scores across different group sizes on MATH.The curves are smoothed with a simple moving average using window size 4 for visualization.
  • D.4 Ablation Study on Group Size: The study reports reward trajectories during training alongside validation scores throughout the training process.
  • D.4 Ablation Study on Group Size: Table 4 provides a comprehensive group-size ablation across models trained on MATH and DAPO and evaluated on five mathematical reasoning benchmarks.

E Limitation

The study focuses on reasoning tasks under RLVR, while extending its contrastive interpretation to continuous rewards only theoretically. The experiments also involve LLMs in both the research process and evaluations.

  • E Limitation: The empirical study focuses primarily on reasoning tasks using verifiable rewards and leaves continuous-reward experiments for future work.The paper states that GRPO’s contrastive nature applies to continuous or binary rewards, but defers continuous-reward evaluation because of space limits.
  • E Limitation: LLMs were used for writing, editing, formatting, and in the experiments.
Loading 2510.00977v3…