PaperlayerView exact revision on arXiv

Source-linked AI summary

The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning

Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, Yu Meng

arXiv:2506.01347v2cs.CLcs.LG

TL;DR

RLVR’s effectiveness depends on how it uses correct and incorrect samples, a mechanism that remains underexplored. The paper decomposes RLVR into PSR and NSR, analyzes their gradients and inference-scaling effects, and finds that NSR-only training is broadly effective while Weighted-REINFORCE improves results across several benchmarks.

  • Problem

    The mechanisms by which RLVR’s correct and incorrect samples shape reasoning and generalization remain underexplored.

  • Method

    The paper decomposes RLVR into PSR and NSR, evaluates them independently across Pass@k, and uses token-level gradient analysis to study their effects.

  • Results

    NSR-only training consistently improves performance across a wide Pass@k spectrum, often matching or outperforming PPO and GRPO, while Weighted-REINFORCE improves Pass@k on MATH, AIME 2025, and AMC23.

  • Takeaways & Limitations

    NSR can preserve generation diversity by suppressing incorrect responses and shifting probability toward plausible alternatives supported by the model prior.

  • Takeaways & Limitations

    The benefits of RL training depend on the backbone: Llama-3.1-8B-Instruct experiences degraded inference-scaling performance, although NSR causes the least degradation.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is a promising approach for training language models (LMs) on reasoning tasks that elicit emergent long chains of thought (CoTs). Unlike supervised learning, it updates the model using both correct and incorrect samples via policy gradients. To better understand its mechanism, we decompose the learning signal into reinforcing correct responses and penalizing incorrect ones, referred to as Positive and Negative Sample Reinforcement (PSR and NSR), respectively. We train Qwen2.5-Math-7B, Qwen3-4B and Llama-3.1-8B-Instruct on a mathematical reasoning dataset and uncover a surprising result: training with only negative samples -- without reinforcing correct responses -- can be highly effective: it consistently improves performance over the base model across the entire Pass@$k$ spectrum $k$ up to 256), often matching or surpassing PPO and GRPO. In contrast, reinforcing only correct responses improves Pass@1 but degrades performance at higher $k$, due to reduced diversity. These inference-scaling trends highlight that solely penalizing incorrect responses may contribute more to performance than previously recognized. Through gradient analysis, we show that NSR works by suppressing incorrect generations and redistributing probability mass toward other plausible candidates, guided by the model's prior beliefs. It refines the model's existing knowledge rather than introducing entirely new behaviors. Building on this insight, we propose a simple variant of the RL objective that upweights NSR, and show that it consistently improves overall Pass@$k$ performance on MATH, AIME 2025, and AMC23. Our code is available at https://github.com/TianHongZXY/RLVR-Decomposed.

1 Introduction

The paper investigates how correct and incorrect samples contribute differently to RLVR, finding that negative-sample training can preserve broad inference-scaling performance while positive-sample training narrows diversity.

  • Motivation: RLVR is decomposed into Positive Sample Reinforcement (PSR) and Negative Sample Reinforcement (NSR) to study their distinct effects on behavior and generalization.The decomposition targets the underexplored roles of correct and incorrect samples in RLVR.
  • Inference scaling: PSR-only training improves Pass@1 but hurts Pass@k at larger k values, indicating reduced output diversity and exploration capacity.This contrasts single-attempt confidence with performance across multiple sampled responses.
  • Inference scaling: NSR-only training consistently improves performance over the base LM across the entire Pass@k spectrum and often matches or surpasses PPO and GRPO.The result is reported for Qwen2.5-Math-7B and Qwen3-4B across inference-time Pass@k evaluation.
  • Mechanism: Gradient analysis indicates that NSR suppresses incorrect reasoning steps while preserving plausible alternatives favored by the model’s prior.The authors characterize NSR as refining existing knowledge rather than aggressively teaching new behaviors.
  • Proposed method: Weighted-REINFORCE upweights NSR and consistently improves Pass@k performance on MATH, AIME 2025, and AMC23 over strong baselines.The method is presented as a simple modification of the REINFORCE objective.

2 RLVR Objective and Decomposition

RLVR uses a binary verifiable reward, and the paper separates its learning signal into updates on correct responses and likelihood reduction on incorrect responses. This decomposition enables independent analysis of how PSR and NSR shape model behavior.

  • RLVR objective: RLVR assigns a shared binary reward to all tokens in a response according to whether the verified answer is correct.The reward is typically +1 for correct responses and −1 for incorrect ones.
  • Decomposition: The paper decomposes the RLVR objective into positive- and negative-sample components to disentangle learning from correct and incorrect responses.The decomposition is motivated by the difficulty of interpreting combined positive and negative updates.
  • Decomposition: PSR increases the likelihood of correct responses, resembling supervised fine-tuning, whereas NSR reduces probability assigned to incorrect responses.Both objectives are defined using responses sampled from the model itself during training.
  • Experimental isolation: The full RLVR objective equals the sum of PSR and NSR, allowing each sub-objective to be trained independently and compared with full RLVR.This setup isolates the individual effects of the two learning paradigms.

3 Positive and Negative Sample Reinforcement for LLM Reasoning

The paper compares PSR and NSR training across models and inference budgets, finding that NSR generally preserves or improves scaling performance while PSR often trades diversity for single-sample accuracy.

  • Experimental Setup: Qwen2.5-Math-7B, Qwen3-4B, and Llama-3.1-8B-Instruct are evaluated with PSR, NSR, PPO, and GRPO on mathematical reasoning tasks.Qwen3-4B uses its non-thinking mode for training and inference; training uses MATH and evaluation uses MATH, AIME 2025, and AMC23.
  • Experimental Setup: Pass@1 reflects exploitation, whereas larger Pass@k values assess diversity and exploration across multiple attempts.The evaluation samples multiple responses and uses an unbiased Pass@k estimator.
  • Inference Scaling Trends: NSR consistently improves over the base model across the Pass@k spectrum and often matches or surpasses PPO and GRPO.For Qwen2.5-Math-7B, NSR is comparable across k values and outperforms the other methods at k = 256.
  • Training Dynamics: NSR suppresses incorrect reasoning steps and redistributes probability mass toward plausible alternatives already favored by the model’s prior.Its higher entropy and preserved uncertainty support exploration, while PSR more rapidly increases correct samples and fully solved prompts.
  • Inference Scaling Trends: PSR improves Pass@1 but eventually falls below the base model at larger k values because it concentrates probability mass on early correct responses.This concentration produces overconfidence, reduced entropy, and a collapsed output distribution that limits diverse correct solutions.
  • Inference Scaling Trends: In Qwen3-4B non-thinking mode, PSR fails to improve the base model, whereas NSR and GRPO improve performance across Pass@k metrics.NSR reaches 94.0 Pass@1 and 98.0 Pass@64 on MATH, closely matching the model’s thinking-mode performance.
  • Inference Scaling Trends: For Llama-3.1-8B-Instruct, all reinforcement-learning methods degrade inference-scaling performance, but NSR causes the least degradation.Although Pass@1 improves on MATH and AMC23, Pass@256 drops substantially after RL training.

4 Understanding the Effectiveness of Negative Sample Reinforcement

The analysis attributes NSR’s effectiveness to preserving diversity while suppressing incorrect reasoning, whereas PSR sharpens distributions and can reduce exploration. Extensions to PPO and GRPO retain these core gradient behaviors.

  • Entropy and inference scaling: NSR maintains high test-set entropy, while PSR rapidly reduces entropy; PPO and GRPO occupy an intermediate range.The entropy pattern helps explain stronger large-k performance for NSR than for methods whose entropy declines.
  • Entropy and inference scaling: NSR improves correct-sample ratios without making the model overconfident, preserving uncertainty that supports Pass@k scaling.PSR instead rapidly increases correct samples and fully solved problems in each batch.
  • Token-level gradient dynamics: PSR increases sampled correct-token logits while decreasing other logits, repeatedly sharpening observed correct sequences and suppressing alternatives.The resulting reduced entropy can produce overfitting and restrict exploration or generalization.
  • Token-level gradient dynamics: NSR demotes sampled incorrect tokens and reallocates probability toward alternative tokens in proportion to their current likelihoods.This prior-guided redistribution promotes plausible alternatives rather than introducing entirely new behaviors.
  • PPO and GRPO: PPO and GRPO add clipping, KL regularization, and advantages, but these modifications preserve the qualitative contrast between positive and negative reinforcement.Clipping preserves update direction, while advantage rescaling or value baselines does not alter the overall gradient dynamics.

5 Balancing Positive and Negative Reinforcement

Weighted-REINFORCE balances PSR’s small-k accuracy with NSR’s diversity by down-weighting positive rewards. Across multiple benchmarks, this simple weighting yields strong Pass@k performance.

  • Method: Weighted-REINFORCE scales down PSR by λ while retaining NSR, balancing accuracy and diversity; λ = 1 recovers vanilla REINFORCE.The method combines learning from correct and incorrect samples with a reduced positive-reward weight.
  • Results: With λ = 0.1, W-REINFORCE matches PPO’s best MATH Pass@1 score of 76.6 and leads for every MATH k ≤64.It remains competitive at k = 256.
  • Results: With λ = 0.1, W-REINFORCE achieves the best AIME 2025 result for every k except k = 1.The experiments use the same moderate positive-reward down-weighting across the reported benchmarks.
  • Results: W-REINFORCE consistently outperforms PPO and GRPO across most Pass@k values, while vanilla REINFORCE underperforms.The authors identify it as a competitive alternative when the model prior is strong, such as for Qwen models.

6 Related Work

The paper addresses gaps in RLVR research by examining inference-time scaling and separating learning from correct versus incorrect responses. It emphasizes negative reinforcement as a way to preserve diversity while improving accuracy.

  • Research gap: Prior RLVR work often emphasizes Pass@1 or greedy decoding, potentially overlooking changes in inference-scaling behavior.The paper also identifies the mechanisms behind reasoning and generalization as underexplored.
  • Approach: The paper decomposes RLVR into learning from correct responses and learning from incorrect responses to study their distinct effects.This decomposition directly targets the entanglement of positive and negative learning signals.
  • Inference-time scaling: Inference-time scaling can improve reasoning performance by generating multiple candidate solutions or longer reasoning traces.This motivates evaluating model behavior beyond single-sample accuracy.
  • Contribution: Negative reinforcement is presented as preserving and improving inference-time scaling, while reward weighting can balance accuracy against diversity.The claim is framed within the paper’s RLVR analysis rather than as a general prescription for all training settings.

7 Conclusion

The paper finds that penalizing incorrect samples can improve reasoning while preserving diversity, and explains this through prior-guided probability redistribution. Weighted-REINFORCE then balances positive and negative reinforcement across benchmarks.

  • Conclusion: NSR-only training improves reasoning across a wide Pass@k spectrum and often matches or exceeds PPO and GRPO.The conclusion frames this as the central empirical finding of the decomposition.
  • Conclusion: Gradient analysis shows NSR suppresses incorrect responses and reallocates probability toward plausible alternatives favored by the model prior.This mechanism refines existing knowledge rather than aggressively teaching new behaviors.
  • Conclusion: Weighted-REINFORCE upweights NSR and achieves a balance between PSR and NSR with consistent Pass@k improvements across multiple reasoning benchmarks.The conclusion identifies this as the paper’s proposed objective variant.

A Gradient Derivation

The section derives the gradient of Equation (6) using simplified notation and omits a constant factor that only scales gradient magnitude.

  • The derivation defines πv as πθ(v|x, y<t) for compact notation.
  • The omitted constant 1/T affects gradient magnitude but not the stated gradient structure.
  • The gradient expression distinguishes the sampled token yt from other vocabulary tokens.

B NSR differs from Entropy and Unlikelihood Training in Gradients

NSR’s gradient dynamics differ from entropy maximization and unlikelihood training: NSR penalizes incorrect outputs using raw probabilities while preserving more of the model’s prior structure.

  • Entropy regularization: Entropy maximization suppresses high-confidence tokens and boosts low-confidence tokens based on their log-probability relative to the vocabulary average.
  • Entropy regularization: Entropy regularization can penalize confidently correct tokens while promoting implausible tokens solely because they are unlikely under the current policy.
  • Unlikelihood training: NSR penalizes incorrect outputs using raw probabilities rather than the −log(1 −πθ(y|x)) objective used by unlikelihood training.
  • Unlikelihood training: Unlikelihood training redistributes mass in proportion to current probabilities, whereas NSR’s (1 −πv) damping produces different update dynamics.
  • RLVR reward structure: RLVR uses binary rewards whose signs remain tied to individual-sequence correctness, enabling separation into positive and negative learning paradigms.

D.1 Objectives and Training Hyperparameters

The appendix presents the compared objectives, implementation settings, prompt-template references, and a λ ablation for Weighted-REINFORCE.

  • Objectives: PPO, GRPO, PSR, and NSR are presented as the objectives used for comparison.
  • Training hyperparameters: PPO and GRPO use KL penalties, whereas PSR and NSR omit them; all objectives receive an entropy bonus of 1e-4.
  • Prompt templates: Qwen prompt templates are referenced in Tables 2 and 3 for the specified Qwen model settings.
  • Training hyperparameters: PSR and NSR disable advantage normalization because normalization would make their raw ±1 reward learning signal zero.
  • λ ablation: Performance remains relatively stable when λ ≤0.2, while larger λ values gradually degrade Pass@k across all k.

F Limitations

The paper identifies instability under extended NSR training and leaves performance with dense, non-binary rewards as an open question.

  • Extended NSR training: Hundreds of NSR gradient steps cause a noticeable performance decline, although comparable degradation also occurs in standard RL algorithms such as GRPO.
  • Extended NSR training: NSR’s prior-preserving mechanism may not ensure stable long-term training, and incorporating some PSR may be necessary.
  • Reward scope: The study focuses on sparse binary outcome rewards, leaving behavior under dense continuous feedback open.
Loading 2506.01347v2…