Source-linked AI summary
Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards
Xinyu Tang, Yuliang Zhan, Zhixun Li, Wayne Xin Zhao, Zhenduo Zhang, Zujie Wen, Zhiqiang Zhang, Jun Zhou
TL;DR
RLVR uses positive and negative rollouts whose distinct effects on reasoning remain insufficiently characterized. This paper analyzes those polarities across granularities and proposes A3PO, which allocates token-level advantages asymmetrically and adaptively. Experiments across models and benchmarks validate its effectiveness while showing complementary sharpening and exploration roles.
Problem
The distinct roles of positive and negative self-generated samples in RLVR remain underexplored.
Method
The paper analyzes polarity effects at sample and token levels and proposes A3PO for adaptive, asymmetric token-level advantage shaping.
Results
Positive samples sharpen correct reasoning patterns, negative samples promote exploration, and experiments across multiple models and benchmarks validate A3PO’s effectiveness.
Takeaways & Limitations
Both sample polarities are essential for RLVR, and more precise advantage allocation can support stable and improved training.
Takeaways & Limitations
Experiments cover only text-based reasoning tasks and do not evaluate agent-based scenarios such as search or code agents.
Abstract
from arXiv · showhide
Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach.
1 Introduction
The paper investigates how positive and negative self-generated samples shape RLVR, finding complementary effects and proposing A3PO for finer-grained advantage allocation.
- Positive samples reinforce correct reasoning paths, whereas negative samples promote self-correction by learning from mistakes.
- Positive samples sharpen existing reasoning, while negative samples increase exploration; relying on either polarity alone impairs reasoning performance.
- Sample- and token-level advantage shaping reveal that positive weighting accelerates reward but narrows exploration, whereas negative weighting broadens exploration more slowly.
- A3PO adaptively adjusts high-probability negative-token and low-probability positive-token advantages for finer-grained allocation.
- The paper reports that experiments across three LLMs and five reasoning benchmarks validate A3PO’s effectiveness.
2 Related Work
Prior RLVR work refines optimization through advantage estimation, loss aggregation, importance sampling, and sampling strategies while increasingly distinguishing positive and negative samples.
- RLVR research has refined advantage estimation, loss aggregation, importance sampling, and sampling strategies.
- Prior methods treat positive and negative samples differently through importance sampling, advantage reweighting, and clipping mechanisms.
- This paper positions its contribution as a more thorough investigation of how sample polarities affect RLVR training.
3 Rethinking the Role of Positive and Negative Samples in RLVR
Across models, positive samples sharpen and compress established reasoning, while negative samples preserve exploration; combining both is essential for robust RLVR training.
- Positive reinforcement rapidly reduces entropy and shortens responses, whereas negative reinforcement maintains higher entropy and produces longer outputs.
- Positive-only or negative-only training damages reasoning ability and boundary, confirming that both sample polarities are essential.
- Negative-sample reinforcement can yield better validation performance despite slower training-reward progress, suggesting a role in preserving generalization.
- For Qwen2.5-7B-Math, combining positive and negative samples accelerates training and improves final reward over either polarity alone.
- Sharpening ranks PSR > DAPO > NSR, whereas discovery ranks NSR > DAPO > PSR, linking positive samples to existing patterns and negative samples to unseen paths.
4 Impacts of Advantage Shaping with Different Sample Polarities at Varying Granularities on RLVR Training
Advantage shaping exposes a polarity-dependent trade-off: positive emphasis sharpens and speeds reward learning, while negative emphasis preserves exploration; token-level effects depend on entropy and probability.
- Higher positive advantages accelerate reward growth but reduce entropy, response length, and exploration diversity.
- Higher negative advantages encourage alternative reasoning paths, increasing entropy and response length while slowing reward improvement.
- The relative positive-to-negative advantage ratio, rather than absolute values, primarily determines training dynamics; balanced ratios support exploration and steady reward improvement.
- Entropy-based shaping shows that high-entropy positive tokens accelerate entropy reduction, while high-probability negative tokens preserve exploration and low-entropy negative tokens accelerate entropy reduction.
- High-probability positive tokens accelerate entropy reduction, sharpening established correct reasoning patterns.
- Reinforcing low-probability positive tokens or high-probability negative tokens raises entropy, whereas reinforcing low-probability negative tokens reduces it.
5 Adaptive and Asymmetric Advantage Shaping for Policy Optimization
A3PO adaptively and asymmetrically reshapes token-level advantages, emphasizing low-probability positive and high-probability negative tokens while decaying these weights during training. It preserves exploration and achieves higher validation accuracy than DAPO despite slightly slower reward growth.
- 5.1 Method: Low-probability positive and high-probability negative tokens maintain higher entropy, encouraging exploration and preventing premature convergence.These token categories motivate A3PO’s adaptive and asymmetric weighting.
- 5.1 Method: The asymmetric weights are gradually reduced as training progresses because retaining them too long can cause engine mismatch and performance collapse.This schedule transitions optimization smoothly toward standard training.
- 5.2 Experiment Setup: Experiments use three language models and compare A3PO with GRPO, DAPO, W-REINFORCE, Fork Tokens, and Lp-Reg baselines.The main-results visual reports performance across reasoning benchmarks and marks statistically significant improvements with an asterisk.
- 5.3 Main Results: A3PO maintains higher entropy and longer responses than DAPO, while its validation-accuracy advantage widens during training.Training reward grows slightly more slowly for A3PO than for DAPO.
- 5.3 Main Results: A3PO dynamically adjusts advantages for high-probability negative tokens and low-probability positive tokens, achieving the best performance among evaluated methods.The method provides finer-grained advantage allocation for more stable and effective RLVR training.
6 Conclusion
The paper finds that positive and negative RLVR samples make distinct contributions: positive samples sharpen correct reasoning, while negative samples promote exploration. Based on this distinction, A3PO allocates advantages more precisely and improves training across multiple models and benchmarks.
- 6 Conclusion: Positive samples sharpen correct reasoning patterns, while negative samples promote exploration, making both sample polarities essential for RLVR training.These findings identify distinct contributions to RLVR training dynamics.
- 6 Conclusion: A3PO uses adaptive and asymmetric token-level advantage shaping to allocate advantages more precisely, leading to stable and improved RLVR training.Experiments across multiple models and benchmarks validate its effectiveness.
7 Limitations
The analysis and method are evaluated only on text-based reasoning tasks and have not been tested in agent-based scenarios because of computational and budget constraints.
- 7 Limitations: The experiments cover only text-based reasoning tasks, leaving applicability to other model families such as vision-language models untested.The authors identify extension to other model families as future work.
- 7 Limitations: Computational and budget constraints prevented evaluation in agent-based scenarios, including search or code agents.
A Detailed Experimental Setup
The experiments decompose RLVR into positive and negative sample reinforcement, then vary polarity-level and token-level advantage weighting to study training dynamics.
- Experimental Setup: Experiments compare PSR, NSR, DAPO, W-REINFORCE, and token-level baselines across three model families using RLVR training and reasoning benchmarks.The setup uses Qwen2.5-Math-7B, Qwen3-8B-Base, and DeepSeek-R1-Distill-Qwen-7B, with DAPO-Math training and evaluation on AIME24, AIME25, Math500, GPQA, and LiveCodeBench.
- Sample Reinforcement: Positive sample reinforcement increases the likelihood of correct responses, whereas negative sample reinforcement reduces the probability of incorrect responses.The RLVR objective is decomposed into correct-rollout and incorrect-rollout sub-objectives to isolate their effects.
- Polarity-Level Shaping: Polarity-level advantage shaping assigns separate scaling factors to positive and negative samples, controlling their relative contributions during policy optimization.βP and βN respectively scale positive- and negative-sample advantages.
- Token-Level Shaping: Token-level advantage shaping reweights selected tokens according to criteria such as entropy and probability to examine their effects on training dynamics.The shaped advantage uses the original group-rollout advantage and a token-level scaling factor for selected tokens.
- Baselines: Baseline comparisons include GRPO, DAPO, W-REINFORCE, DAPO with Fork Tokens, and Lp-Reg.These methods respectively provide group-relative, enhanced, polarity-level, high-entropy-token, and exploration-preserving comparison points.
D Different Training Dynamics of Base LLMs
Training dynamics differ across base LLMs and sample polarities: positive reinforcement sharpens familiar patterns, while negative reinforcement supports broader exploration but can produce slower or unstable progress.
- Analysis Protocol: The validation analysis monitors PSR, NSR, and DAPO across three models and tracks five accuracy-change patterns during training.Figures 8–10 report the corresponding dynamics for Qwen2.5-7B-Math, Qwen3-8B-Base, and DeepSeek-R1-Distilled-Qwen-7B.
- Base-Model Dynamics: Qwen2.5-7B-Math shows more sharpened than degraded validation samples under both PSR and NSR, with combined polarities providing a complementary boost.The analysis categorizes validation accuracy changes as sharpen, degradation, fluctuation, mastery, or struggle using k=10.
- Base-Model Dynamics: Qwen3-8B-Base experiences reward hacking under positive-only reinforcement, while negative reinforcement supports different training behavior.Continuous positive reinforcement progressively shortens responses until the model outputs only the final answer without step-by-step reasoning.
- Ratio Sensitivity: A 20% token-shaping ratio is the main setting, with additional experiments at 5%, 10%, and 50% under a fixed 0.2× scaling factor.Smaller ratios accelerate early entropy reduction and reward improvement but do not change the overall learning trend.
G Negative Samples Amplify the Training-Inference Mismatch
Negative samples widen the training–inference probability gap, and increasing their advantage further amplifies this mismatch; A3PO responds with adaptive weighting for stable training.
- Mismatch Analysis: Higher negative-sample advantages further increase the training–inference probability difference, beyond the widening associated with using negative samples.The comparison covers PSR, NSR, and DAPO, with polarity-level shaping applied to negative samples.
- Adaptive Strategy: A3PO increases the weight of high-probability negative samples early, then decays it until matching the positive-sample weight.This schedule is designed to promote exploration initially while ensuring stable training later.
- Ablation Study: Removing any A3PO component degrades performance on three math benchmarks, while positive low-probability and negative high-probability shaping improve performance.The ablation also reports that omitting the adaptive strategy causes training instability from training–inference mismatch, limiting further AIME25 improvements.
- Robustness: A3PO consistently achieves the best performance across varying model scales and training datasets in experiments on DeepSeek-R1-Distilled-Qwen-7B.The result is reported as evidence of effectiveness and generalizability across those scales and datasets.
H.3 Hyperparameter Analysis
Hyperparameter and auxiliary analyses examine token-selection ratios, scaling schedules, engine mismatch, model and dataset variation, and polarity-specific training dynamics.
- Robustness and Mismatch: Figure 14 evaluates the proposed method across different LLM scales and training datasets, while Figure 13 examines training–inference token-probability differences.Figure 21 and Figure 22 focus specifically on positive- and negative-sample polarity-level shaping.
- Hyperparameter Analysis: A 20% token-shaped ratio achieves optimal performance; lower ratios insufficiently explore tokens, while higher ratios shape too many less relevant tokens.The analysis also studies initial scaling factors ρ and decay coefficients α, with appropriate ρ values important for stable training.
- Ratio Sensitivity: Figure 31 tests different token-shaped ratios for low-probability positive tokens when their advantages are scaled by 0.2.This complements the main ratio-sensitivity analysis.