PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting

Yunzhen Feng, Parag Jain, Anthony Hartshorn, Yaqi Duan, Julia Kempe

arXiv:2510.08696v1cs.LG

TL;DR

GRPO groups with no correct sampled response yield zero advantage and no gradient, wasting compute, particularly early in training and on harder reasoning problems. LENS derives a modified value function from the MLE gradient, adding a confidence-weighted penalty to incorrect responses and assigning them non-zero rewards in GRPO. Our GRPO variant consistently outperforms the GRPO baseline across all Pass@k metrics on Qwen-2.5-3B-Base, with gains concentrated on Levels 4-5.

  • Problem

    GRPO groups with no correct sampled response yield zero advantage and no gradient, wasting compute, particularly early in training and on harder reasoning problems.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    LENS derives a modified value function from the MLE gradient, adding a confidence-weighted penalty to incorrect responses and assigning them non-zero rewards in GRPO.

    Source: §1 · pp. 1–2

  • Results

    Our GRPO variant consistently outperforms the GRPO baseline across all Pass@k metrics on Qwen-2.5-3B-Base, with gains concentrated on Levels 4-5.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework provides a calibrated reward that fits into GRPO and makes negative groups useful for gradient updates.

    Source: §1 · p. 1, §7 · p. 10

    The empirical study uses the simple case; preference-aware variants and systematic experiments with nonbinary rewards are left for future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a standard recipe for improving large language models (LLMs) on reasoning tasks, with Group Relative Policy Optimization (GRPO) widely used in practice. Yet GRPO wastes substantial compute on negative groups: groups in which no sampled response is correct yield zero advantage and thus no gradient. We ask whether negative groups can be leveraged without extra supervision. Starting from a maximum-likelihood (MLE) objective in reward modeling, we show that the MLE gradient is equivalent to a policy gradient for a modified value function. This value function adds a confidence-weighted penalty on incorrect responses, imposing larger penalties on more confident mistakes. We refer to this as \textbf{L}ikelihood \textbf{E}stimation with \textbf{N}egative \textbf{S}amples (\textbf{LENS}). LENS modifies GRPO to assign non-zero, confidence-dependent rewards to incorrect generations, making negative groups informative and converting previously wasted samples into useful gradient updates. On the MATH benchmark with Llama-3.1-8B and Qwen-2.5-3B, the proposed variant consistently outperforms GRPO baseline, with significant gains on harder items. These results demonstrate a principled and practical way to "rescue" negative groups, improving efficiency and performance in RLVR.

1 Introduction

GRPO receives no gradient from groups in which every sampled answer is incorrect, a waste that is common early in training and on harder problems. LENS derives a confidence-weighted penalty from an MLE-to-policy-gradient connection and uses it to make those groups informative; on MATH, it outperforms GRPO across Pass@k in both evaluated model settings, with gains concentrated on harder items.

1 Introduction

  • GRPO assigns zero advantage to groups with no correct sampled response, leaving their generated trajectories without a gradient signal.The paper identifies this inefficiency as especially common early in training and on harder reasoning problems.Source: §1 · p. 1
  • The MLE gradient is equivalent to a policy gradient for a modified value function that penalizes incorrect responses more when the model is more confident.The penalty is intended to discourage overconfident failures and encourage exploration of lower-probability plausible alternatives.Source: §1 · p. 1
  • LENS assigns incorrect generations non-zero, confidence-dependent rewards, making negative groups informative and converting discarded samples into gradient updates.The method is presented as a drop-in GRPO change with negligible computational overhead.Source: §1 · pp. 1–2
  • LENS consistently outperforms GRPO across all Pass@k metrics with Llama-3.1-8B-Instruct and Qwen-2.5-3B-Base, with gains concentrated on MATH Levels 4–5.The reported pattern holds across both evaluated model settings.Source: §1 · pp. 1–2
  • The paper frames LENS as a likelihood framework connecting reward modeling with policy optimization and calibrating how negative samples contribute to the objective.The overview contrasts LENS's confidence-dependent rewards with GRPO's uniform zero reward for incorrect answers.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work studies RLVR and GRPO, as well as ways to use negative samples and information within incorrect trajectories. LENS contributes a framework that stratifies negative-sample rewards using outcome rewards and probability.

2 Related Work

  • RLVR uses algorithmically verified outcomes, and GRPO computes advantages from grouped outputs; an all-incorrect group has zero advantage and does not contribute to optimization.The cited work situates GRPO as an efficient variant of PPO that omits a learned value network.Source: §2 · p. 2
  • Prior approaches reweight positive and negative samples, train with negative-only rewards, or adjust negative-sample signals for off-policy data (Roux et al., 2025; Arnal et al., 2025; Lyu et al., 2025; Zhu et al., 2025).These works explore asymmetric signals and token-level or sample-level reweighting.Source: §2 · p. 3
  • Other approaches extract fractional rewards, correct sub-steps, or negative rationales from wrong completions (Chen et al., 2025a; Yang et al., 2025; Li et al., 2024b).The passage notes that evaluating intermediate reasoning steps is labor-intensive and accurate automation remains underexplored.Source: §2 · p. 3
  • LENS stratifies reward signals within negative samples using outcome rewards and probability, aiming to balance computational efficiency with learning from structured negatives.Source: §2 · p. 3

3 Preliminaries and Motivation

The preliminaries define reasoning accuracy as expected verifiable reward and describe how GRPO estimates policy updates from grouped generations. The motivation is that all-negative groups consume generation compute but provide no learning signal.

3.1 Language Model Reasoning as Policy Optimization

  • The policy-optimization setup treats the language model as generating an answer evaluated by a binary correctness reward, with training aimed at maximizing expected reward over questions.The question distribution is denoted ξ.Source: §3.1 · p. 3

3.2 Motivation: Negative Groups in RLVR

  • GRPO samples a group of candidate answers per question, uses their scalar rewards to form group-relative advantages, and applies the same outcome-derived advantage to every token in a response.Its practical appeal is computing advantages from group statistics rather than a learned value function.Source: §3.2 · pp. 3–4
  • The GRPO objective uses a clipped PPO-style surrogate with a per-token KL regularizer; the described implementation omits the KL term by setting β = 0.Source: §3.2 · pp. 3–4
  • GRPO's all-negative groups provide no policy-gradient contribution; with group size 16, nearly 45% of groups are all-negative early in training and about 35% remain by the end.These groups consume generation compute despite contributing no learning signal.Source: §3.2 · p. 4

4 A Likelihood-Based Framework for Reasoning

The section connects reward modeling to policy optimization, then uses the MLE gradient to motivate calibrated rewards for incorrect answers. It interprets that gradient as optimizing expected reward minus a confidence-weighted penalty, making confident mistakes costly.

4 A Likelihood-Based Framework for Reasoning

  • Reward modeling provides a likelihood-based route to using negative groups and connects the estimation of reward probabilities to policy optimization.The framework aims to learn from negative groups by fitting a model that distinguishes correct from incorrect responses.Source: §4 · p. 4, §4.1 · p. 5

4.1 From Policy Learning to Reward Modeling

  • Estimating reward probabilities p⋆ from observed responses provides a way to recover an optimal policy π⋆ and reframe policy learning as statistical estimation.The paper illustrates the relationship with a multiple-choice setting and notes that preferences among correct answers can also be incorporated.Source: §4.1 · pp. 4–5

4.2 Calibrating Policy Gradient via MLE.

  • The MLE gradient contains a negative-sample contribution beyond the standard positive-sample policy-gradient signal, motivating calibrated rewards for incorrect responses.Incorrect samples receive an adjustment based on policy confidence; correct samples retain reward 1, and negative groups can therefore contribute informative gradients.Source: §4.2 · p. 6
  • Under correct model specification, the true parameter maximizes the population log-likelihood.The paper states this consistency result for the case where the true policy belongs to the parameterized policy class.Source: §4.2 · p. 6

4.3 Confidence Weighted Value Function

  • The MLE gradient equals the gradient of a modified value function that maximizes expected reward while penalizing incorrect responses according to confidence.The penalty is strongest for confident mistakes and negligible for uncertain incorrect responses.Source: §4.3 · pp. 6–7

5 Proposed Modification to GRPO

The proposed GRPO modification substitutes a calibrated reward for the original reward while leaving the GRPO loss unchanged. Its implementation handles group normalization and confidence-based signals, including length-normalized probabilities and a down-weighted contribution from negative groups.

5 Proposed Modification to GRPO

  • The method replaces GRPO’s original reward with a calibrated reward, normalizes it, and uses the resulting advantage without changing the GRPO loss.The calibrated reward implements the gradient of the likelihood-based objective.Source: §5 · p. 7

5.1 Implementation and Practical Considerations

  • Length-normalized geometric-mean probabilities mitigate sequence-length decay that would otherwise make the reward adjustment sparse.The passage explains that raw sequence probabilities decay roughly as γ^|o| for long generations.Source: §5.1 · p. 7
  • The framework can incorporate preferences among correct generations, with the described substitution yielding calibrated rewards that encode a length preference.The passage gives a preference for shorter correct answers as an example.Source: §5.1 · p. 7
  • Smaller difficulty values D(q) increase the penalty on confident incorrect predictions and encourage broader exploration.The true difficulty function is unknown and controls learning dynamics.Source: §5.1 · p. 7
  • Scaling negative calibrated rewards by 1/G preserves GRPO’s sign invariance in the extreme mixed group with one correct generation.The example calibrated rewards are [1, 0, −1, …, −1].Source: §5.1 · p. 8
  • Negative groups use de-meaning-only normalization and a single hyperparameter α to down-weight their weaker, confidence-difference signal.In these groups, there is no verifiable reward signal.Source: §5.1 · p. 8

6 Experimental Results

On MATH, LENS improves over GRPO on both models, with particularly strong gains on hard items. The experiments also report higher Pass@k and improvements from negative groups alone.

Set-up

  • The evaluation uses Llama-3.1-8B-Instruct and Qwen-2.5-3B-Base, training on MATH combined with either DAPO or Numina 1.5.The DAPO setting uses two independent runs; evaluation is on the MATH test set.Source: §6 · p. 8

Results

  • On MATH + DAPO, Figure 5 shows LENS outperforming GRPO on the full test set and Levels 4–5 for both models.The curves average 16 evaluation generations and two independent runs; training remains stable for more than 1,000 steps, and Numina results show consistent improvements with identical hyperparameters.Source: Figure 5 · p. 9, §6 · p. 9

Training protocol and Evaluation

  • Training samples 16 completions per question and uses 512 trajectories per gradient update; evaluation reports Pass@k for k=1, 2, 4, 8, and 16.Evaluation curves cover the full MATH test set and its Levels 4–5 subset.Source: §6 · p. 9

Results

  • LENS achieves higher Pass@k than GRPO across k=1, 2, 4, 8, and 16, with the Pass@16 improvement also significant.Table 1 reports Pass@k on MATH for Llama-3.1-8B-Instruct and Qwen-2.5-3B-Base, averaged over two runs on MATH + DAPO.Source: §6 · p. 9, Table 1 · p. 10
  • Ablations show strong improvements from adjusted rewards in negative groups alone.Source: §6 · p. 10

7 Discussion

The discussion links reward-model likelihood estimation to policy optimization and reports improvements across Pass@k on both evaluated models. It identifies empirical extensions and hyperparameter-free objectives as future work, while distinguishing LENS from related approaches to exploration and negative groups.

7 Discussion

  • The discussion presents LENS as connecting reward-modeling MLE to policy optimization through a confidence-penalized value function, with improvements across Pass@k on both models.Source: §7 · p. 10
  • The empirical method uses a tunable hyperparameter to balance negative and mixed groups; systematic experiments with nonbinary rewards and preference-aware variants remain future work.Source: §7 · p. 10

A Other Related Works

  • The authors contrast LENS with confidence-proxy methods that replace rule-based rewards, reporting instead stable training beyond 1,500 steps on both Llama and Qwen.Source: A Other Related Works · p. 14
  • Related negative-group approaches use baselines, advantage calibration, entropy, or more sampling; the authors distinguish LENS by its theory-grounded motivation and reward differentiation within negative groups.Source: A Other Related Works · p. 14

B.1 Proof of Theorem 1

The appendix derives the log-likelihood gradient and establishes its consistency under correct specification, then proves that the population log-likelihood gradient matches the gradient of the proposed value function.

B.1 Proof of Theorem 1

  • The chain rule and the gradient-of-a-logarithm identity yield the individual loss-term gradient used in the proof.The proof applies ∇θπθ = πθ · ∇θlogπθ before reaching its final result.Source: §B.1 · p. 14
  • Under correct specification, the true parameter is a population log-likelihood maximizer because the expected gradient is zero there.The conditional expectation of the gradient’s inner term is zero, making θ⋆ a stationary point.Source: §B.1 · p. 15

B.2 Proof of Theorem 2

  • When samples are drawn from πθ, the population log-likelihood gradient is identical to the gradient of JMLE(πθ).The proof combines the positive and negative objective gradients and identifies the result with the MLE gradient.Source: §B.2 · p. 15

C A Preference-Aware Framework

The preference-aware framework generalizes the uniform target distribution by weighting question–answer pairs through a nonnegative preference function. It describes alternatives based on data collection, the policy, or response length, and specifies training settings.

C A Preference-Aware Framework

  • The framework replaces the uniform distribution over correct answers with preferences that can favor answers for reasons such as ease, brevity, or naturalness.A preference function ρ(q, o) adjusts the weight assigned to each question–answer pair and adds a signal to both positive and negative samples.Source: C A Preference-Aware Framework · pp. 15–16
  • With preferences, pθ(q, o) incorporates ρ(q, o) and the difficulty factor D(q), which accounts for internal weighting across candidate answers.The framework also formulates an MLE problem under this modified relation between policy and correctness probabilities.Source: C A Preference-Aware Framework · p. 16
  • Using the data-collection distribution as preference makes D(q) estimable from observed answers, while using the policy itself makes negative-sample weights equal its correction rate.For data-collection preferences, D(q) is approximated by the inverse empirical correctness rate; policy preference yields uniform weights as a special case.Source: C A Preference-Aware Framework · pp. 16–17

D.1 Hyperparameters

  • In practice, it is convenient to set the negative-sample reward with γ > 0 properly tuned; training uses learning rates of 3e−7 for Llama-3.1-8B-Instruct and 1e−6 for Qwen-2.5-3B-Base, with batch size 512.The batch comprises 32 questions and 16 generations each. The training set combines 7,500 MATH samples with a random subset of 17,500 DAPO samples.Source: C A Preference-Aware Framework · p. 17, §D.1 · p. 17

D.2 Ablation

The ablation reports that adjusting incorrect-response rewards in mixed and negative groups both helps, with negative-only LENS improving over GRPO and sometimes nearly matching full LENS.

D.2 Ablation

  • Both reward-adjustment components help improve performance.The ablation evaluates pass@k on MATH with Llama-3.1-8B-Instruct.Source: Table 2 · p. 18
  • The ablation isolates mixed-group reward changes from negative-group reward changes using two LENS variants.One variant modifies incorrect generations in mixed groups while keeping negative-group advantages at zero; the other modifies negative groups while leaving mixed groups unchanged.Source: §D.2 · pp. 17–18
  • Negative-only LENS improves over GRPO and in several cases nearly matches full LENS, indicating that negative groups account for a substantial portion of the improvement.Mixed-only LENS has its largest gains at pass@1; adjusting mixed-group rewards supports reinforcing correct samples while rejecting incorrect ones.Source: §D.2 · p. 18, Table 2 · p. 18

E Additional Results

Additional results on MATH and Numina 1.5 report higher Pass@k for larger k, indicating greater diversity, and training curves show improvement for both models on the full test set and hard subset.

E Additional Results

  • The additional MATH-plus-Numina results are complementary to the main paper’s results and were omitted there for space.Only one training run was performed because of computational constraints.Source: E Additional Results · p. 18
  • For MATH, Pass@k significantly improves for larger k, indicating greater diversity; the reported result comes from one training run.Table 3 reports Pass@k results for Llama-3.1-8B-Instruct and Qwen-2.5-3B-Base.Source: E Additional Results · p. 18, Table 3 · p. 18
  • Figure 6 shows improvement over GRPO for both models on the full MATH test set and Levels 4–5 hard subset.Its accuracy averages all 16 generations during evaluation; training uses MATH and Numina 1.5.Source: Figure 6 · p. 19

Related papers

Loading 2510.08696v1…