PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs

Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, Mao Yang

arXiv:2506.14245v2cs.AIcs.CL

TL;DR

The paper addresses whether RLVR genuinely expands LLM reasoning or merely improves sampling efficiency. It combines benchmark evaluation, CoT-Pass@K, theoretical analysis, and training-dynamics studies, finding extended reasoning boundaries and improved CoT quality across math and code tasks. The authors conclude that RLVR implicitly incentivizes correct reasoning, while noting verifier and generalization limitations.

  • Problem

    The paper asks whether RLVR genuinely improves reasoning beyond sampling efficiency, prompted by Pass@K evidence that base models can catch up with post-RLVR models.

  • Method

    The paper combines Pass@K and CoT-Pass@K evaluations, a theoretical GRPO analysis, training-dynamics experiments, and learning-based CoT-quality evaluation.

  • Results

    RLVR extends reasoning capability boundaries in mathematical and coding tasks and fundamentally improves the quality of generated reasoning CoTs.

  • Takeaways & Limitations

    The findings support the perspective that RLVR implicitly incentivizes correct reasoning rather than only improving sampling efficiency.

  • Takeaways & Limitations

    CoT correctness relies on an LLM verifier, and the theorem explains RLVR optimization but does not guarantee generalization.

Abstract

from arXiv · show

Recent advancements in long chain-of-thought (CoT) reasoning, particularly through the Group Relative Policy Optimization algorithm used by DeepSeek-R1, have led to significant interest in the potential of Reinforcement Learning with Verifiable Rewards (RLVR) for Large Language Models (LLMs). While RLVR promises to improve reasoning by allowing models to learn from free exploration, there remains debate over whether it truly enhances reasoning abilities or simply boosts sampling efficiency. This paper systematically investigates the impact of RLVR on LLM reasoning. We revisit Pass@K experiments and demonstrate that RLVR can extend the reasoning boundary for both mathematical and coding tasks. This is supported by our introduction of a novel evaluation metric, CoT-Pass@K, which captures reasoning success by accounting for both the final answer and intermediate reasoning steps. Furthermore, we present a theoretical framework explaining RLVR's incentive mechanism, demonstrating how it can encourage correct reasoning even when rewards are based solely on answer correctness. Our analysis of RLVR's training dynamics reveals that it incentivizes correct reasoning early in the process, with substantial improvements in reasoning quality confirmed through extensive evaluations. These findings provide strong evidence of RLVR's potential to enhance LLM reasoning, offering valuable insights into its mechanisms and performance improvements.

1 INTRODUCTION

The paper examines whether RLVR expands LLM reasoning or only improves sampling efficiency. It argues that RLVR extends reasoning boundaries, incentivizes correct reasoning, and improves CoT quality.

  • RLVR research grew after DeepSeek-R1 demonstrated long-CoT reasoning with GRPO.The paradigm uses verifiable rewards to train LLM policies through exploration.
  • Pass@K findings raised doubts because base models can catch up with or surpass post-RLVR models as K increases.This motivated the hypothesis that RLVR changes sampling efficiency without expanding reasoning capacity.
  • Prior studies reported conflicting evidence about whether RLVR creates new reasoning patterns or improves reasoning boundaries.Reported effects differ across mathematical and coding tasks.
  • CoT-Pass@K identifies extended mathematical reasoning boundaries by requiring both correct answers and correct intermediate reasoning.The metric addresses cases where incorrect CoTs reach the ground truth by coincidence.
  • A theoretical framework explains that GRPO increases the probability of correct CoTs when pretraining supplies priors distinguishing correct from incorrect reasoning.The framework applies even when rewards encode only answer correctness.
  • Training analysis finds that RLVR incentivizes correct reasoning early, generalizes to unseen questions, and improves the quality of generated CoTs.The paper also reports that improved CoTs can reproduce post-RLVR generalization through supervised fine-tuning.

2 RELATED WORK

Related work centers on whether RLVR improves reasoning beyond base-model sampling efficiency and on how reasoning-chain correctness should be evaluated. The paper extends prior empirical comparisons across math and coding settings.

  • RLVR: RLVR studies examine training data, objectives, model behavior, and reasoning improvements, usually in models up to 32B parameters.The literature expanded rapidly after DeepSeek-R1, partly because RLVR is computationally expensive.
  • Debate on RLVR: Yue et al. proposed that all correct reasoning paths already exist in base models and RLVR mainly improves sampling efficiency.Their view was motivated by base models catching up on Pass@K at moderately large K.
  • Debate on RLVR: Other studies reported new reasoning patterns, persistent coding gains, or task-dependent differences in Pass@K outcomes.These findings leave the relationship between RLVR and reasoning-capacity expansion unsettled.
  • Coding evidence: Earlier coding studies found extended reasoning boundaries after RLVR, including when training begins from distilled LLMs.The present work broadens these comparisons across additional LiveCodeBench versions.
  • The Importance of Correct CoTs: Prior CoT-verification work often uses synthetic tasks, while unstructured reasoning scenarios remain harder to evaluate reliably.This motivates evaluating detailed reasoning correctness in realistic mathematical tasks.

3 EXTENDED REASONING CAPABILITY BOUNDARY AFTER RLVR

Benchmark evaluations indicate that RLVR can extend reasoning boundaries in mathematics and competitive coding, beyond improving sampling efficiency. Reliable evaluation and challenging, uncontaminated benchmarks are important for exposing these gains.

  • The evaluation compares RLVR models with base or distilled counterparts using Pass@K and, for math, CoT-Pass@K.The study emphasizes training recipes, challenging benchmarks, and reliable metrics.
  • Math reasoning: CoT-Pass@K verifies mathematical reasoning chains with repeated LLM judgments using any-correct, all-correct, and majority-correct strategies.DeepSeek-R1-0528-Qwen3-8B serves as the lightweight verifier.
  • Math reasoning: On AIME 2024 and AIME 2025, DAPO-Qwen-32B maintains a significant CoT-Pass@K advantage over Qwen2.5-32B through K=1024.Ordinary Pass@K instead shows the base model catching up with or surpassing the post-RLVR model as K grows.
  • Math reasoning: RLVR effects are weaker on MATH-500 and AMC23 because the base model can solve them with enough attempts, while Minerva shows no improvement.The Minerva result is attributed to mismatch between physics/free-form evaluation and integer-answer math training.
  • Code reasoning: Pass@K is reliable for code because executing generated snippets verifies correctness and reduces guessing.This differs from mathematical evaluation based on answer-token correctness.
  • Code reasoning: AceReason-Nemotron-7B improves Pass@K over DeepSeek-R1-Distill-Qwen-7B on most LiveCodeBench versions.The results indicate that high-quality RLVR can extend the boundary even for already capable distilled models.
  • Code reasoning: Skywork-OR1 shows a consistent Pass@K gap over DeepSeek-R1-Distill-Qwen-7B on LiveCodeBench-v6, driven by medium and hard problems at large K.The comparison underscores the value of challenging benchmarks.

4 A THEORETICAL UNDERSTANDING OF RLVR FOR LLMS

The paper explains how answer-only RLVR rewards can implicitly favor correct reasoning in pretrained LLMs, while identifying assumptions and failure modes that constrain this mechanism.

  • Motivation and setup: RLVR for pretrained LLMs differs from traditional RL because models can generate reasoning chains that sometimes reach the correct answer through imperfect reasoning.The framework separates chain-of-thought correctness from final-answer correctness, allowing coincidental answers to be analyzed.
  • Logic Prior assumption: The framework assumes correct CoTs are more likely than incorrect CoTs to induce correct answers, expressed as α > β.This Logic Prior is attributed to pretrained models’ established knowledge and logic priors.
  • Theoretical result: Theorem 1 states that the expected GRPO advantage and policy gradient increase the probability of generating correct CoTs under the stated assumptions.The result holds with a learnable group and sufficiently large sampling number G for stable updates.
  • Mechanism: As training improves sound reasoning and reduces spurious correlations, the α − β gap widens and accelerates coherent reasoning.The discussion describes correct-CoT probability increasing while incorrect-CoT effects decrease during training.
  • Failure modes: The Logic Prior may fail when pretrained biases or knowledge errors occur in CoTs that still produce correct answers, potentially reinforcing those incorrect reasoning patterns.This is identified as a failure mode of answer-correctness-only reinforcement.

5 TRAINING DYNAMICS OF RLVR

Training analysis of DAPO shows that RLVR improves answer and reasoning quality from early training, generalizes to unseen questions, and exposes a saturation-related limitation.

  • Key indicators: The analysis tracks answer passes and joint CoT-answer passes for sampled responses, then averages per-prompt metrics across datasets.These indicators support estimates of answer correctness and correct reasoning among answer-passing responses.
  • Optimization effects: On fully optimized training questions, correct-answer probability approaches 1 while P(CC|CA)(q) also improves, supporting implicit incentives for correct reasoning.The observed changes align with the theoretical prediction that answer-only rewards can improve reasoning CoTs.
  • Generalization behaviors: RLVR training improves both Pass@K and CoT-Pass@K on AIME 2024 and 2025 from the beginning, indicating an early expansion of the reasoning capability boundary.CoT-Pass@K incorporates reasoning correctness rather than only final-answer success.
  • Generalization behaviors: The training improvements generalize to unseen AIME test questions, despite the absence of formal supervision for CoT correctness.The analysis interprets this as implicitly learned ability rather than directly supervised reasoning verification.
  • Limitations of DAPO: After 400 steps, most fully optimized questions approach P(CA)(q)=1.0 and become unlearnable for GRPO, while median P(CC|CA)(q) remains around 0.7.All-correct groups provide no valid GRPO advantage, leaving imperfect CoTs that answer-only rewards may not mitigate.

6 THE QUALITY OF REASONING COTS ENHANCED BY RLVR

The paper evaluates RLVR-enhanced CoTs through supervised fine-tuning and finds that later-stage CoT data produces stronger generalization, while noting verifier and theoretical-scope limitations.

  • Evaluation approach: The study assesses CoT quality by training identical base models on CoTs from different checkpoints and measuring post-SFT generalization on AIME test sets.Pass@1, Pass@K, and CoT-Pass@K serve as proxies for the quality of the CoT data.
  • CoT quality during RLVR: As RLVR progresses, post-SFT Pass@1 improves steadily, and SFT on DAPO CoT data ultimately matches DAPO-Qwen-32B’s Pass@1.This indicates that later-stage CoT data can reproduce nearly the same Pass@1 performance through SFT.
  • CoT quality during RLVR: Even CoTs containing identifiable errors improve in overall quality during later RLVR stages, as measured by Pass@1 generalization.The result suggests that residual erroneous steps do not preclude broader quality gains in the generated reasoning data.
  • Before-and-after RLVR: SFT on DAPO CoT data nearly replicates post-RLVR performance, whereas SFT on base-model CoTs begins to mitigate guessing when only correct-answer CoTs are used.The latter procedure is interpreted as a round of off-policy RLVR optimization.
  • Limitations: The study relies on an LLM verifier for CoT correctness, and its theorem explains optimization but does not guarantee generalization.Generalization claims are therefore supported empirically rather than by the theorem.

7 CONCLUSION

The paper argues that RLVR implicitly incentivizes correct reasoning, supported by empirical evaluations and theoretical analysis. It further connects post-RLVR reasoning improvements with capabilities that supervised learning can replicate.

  • Empirical evaluations and theoretical analysis support the perspective that RLVR implicitly incentivizes correct reasoning.
  • Training-dynamics and CoT-quality analyses indicate that reasoning CoTs after RLVR are fundamentally different.
  • Post-RLVR reasoning CoTs can help replicate similar capabilities through supervised learning.
  • The authors envision RLVR as a cornerstone for LLMs that learn through interaction, self-correction, and verifiable reasoning.

A.1 CONTRIBUTIONS

This section introduces the paper’s contributions and identifies the researchers responsible for experiments, analysis, theoretical development, tools, and manuscript preparation.

  • The section presents a consolidated list of the paper’s individual contributions.
  • Xumeng Wen led rollout, verification, and mathematical Pass@K analysis, while Zihan Liu examined mathematical reasoning cases and CoT-Judge reliability.
  • Shun Zheng designed the research, conducted RLVR experiments, identified CoT correctness as key, built the theory, and wrote the manuscript.
  • Shengyu Ye developed the EvalHub tool.

A.2 DATA SOURCES

The paper draws its mathematical evaluations from several established benchmarks and reuses DAPO’s training and evaluation data and prompt template. Competitive-coding Pass@K evaluations follow the official LiveCodeBench repository.

  • Mathematical benchmarks include AIME 2025, AIME 2024, Math-500, AMC23, and Minerva.
  • DAPO training and evaluation reuse the DAPO-Math-17k dataset and processed AIME 2024 data.
  • The study reuses DAPO’s prompt template and notes that its training questions were duplicated for compatibility with an old VERL version.
  • Competitive-coding Pass@K evaluations follow the official LiveCodeBench repository.

A.3 LLM-AS-A-COT-JUDGE FOR MATH REASONING

The paper uses repeated specialized-LLM verification to assess the correctness of mathematical reasoning CoTs, aggregating results with any-correct, all-correct, and majority-correct strategies. This design addresses verifier errors by balancing false-positive and false-negative control.

  • DeepSeek-R1-0528-Qwen3-8B verifies mathematical CoTs generated by Qwen2.5-32B and is used as a specialized reasoning verifier.
  • Each CoT receives independent verification under any-correct, all-correct, and majority-correct aggregation strategies.
  • Majority-correct classifies a CoT according to the majority of verification outcomes.
  • Any-correct accepts a CoT when at least one verification passes, while all-correct requires every verification to pass.
  • For n independent attempts, all-correct reduces false positives exponentially and any-correct reduces false negatives exponentially.
  • The study uses n = 3 verification attempts for each CoT.
  • The verifier prompt emphasizes step-by-step reasoning analysis rather than only checking the final answer.

A.4 REVISITING PASS@K EXPERIMENTS FOR SKYWORK-OR1

On LiveCodeBench-v6, RLVR improves both sampling efficiency and the reasoning boundary for distilled models, whereas on math tasks its gains appear concentrated in sampling efficiency.

  • Code tasks: Skywork-OR1-7B significantly improves both Pass@1 and Pass@K up to K=1024 over DeepSeek-R1-Distill-Qwen-7B on LiveCodeBench-v6.The comparison covers LiveCodeBench-v6 and its difficulty-level subsets.
  • Math tasks: Even with CoT-Pass@K, the post-RLVR and distilled math models show no distinct Pass@K gap at large K values.The reported math improvements therefore appear mainly in Pass@1.
  • Cross-domain interpretation: For code domains, RLVR can stimulate distilled models to fit real-world execution feedback, incentivizing an extended reasoning boundary.This contrasts with the reported math-domain pattern, where major reasoning capabilities may already be mastered by the distilled model.
  • Math tasks: On AIME 2024 and 2025, Figure 9 compares Pass@K and CoT-Pass@K between Skywork-OR1-Math-7B and the distilled baseline.CoT-Pass@K uses multiple verifications per chain of thought with any-correct, all-correct, and majority-correct strategies.

A.5 PROOF AND ADDITIONAL THEORETICAL ANALYSIS

The paper combines theoretical analysis, training-dynamics evidence, and case studies to argue that RLVR can promote correct reasoning, while identifying verification, benchmark, and optimization limitations.

  • Proof and theoretical analysis: The proof develops expected-reward and advantage expressions for analyzing how GRPO updates depend on CoT correctness.The derivation defines the current probability of generating a correct CoT and conditions expected advantages on CoT correctness.
  • Training dynamics: RLVR begins incentivizing correct reasoning early, with CoT-Pass@K gains on AIME 2024 within the first 20 training steps.Each training step contains 16 gradient updates in the DAPO script, so generalization begins after only a few gradient updates.
  • Training dynamics: P(CC|CA)(q) rises steadily for easy and hard questions, while P(CA)(q) can misleadingly approach 1 despite flawed reasoning.After 180 steps, the P(CA)(q) distribution for easy questions becomes highly skewed toward 1.0, whereas many responses still contain incorrect CoTs.
  • Limitations: Improving correct CoTs remains slow and challenging for both easy and hard training questions.The paper identifies accelerating P(CC|CA)(q) improvement as a direction for improving RLVR efficiency and effectiveness.
  • Limitations: Static benchmarks face contamination risks that can undermine the reliability of observed improvements.The discussion calls for live, evolving, challenging benchmarks.
  • Limitations: The CoT verifier is useful but not infallible, motivating multiple verification strategies and lighter reliable evaluators.Conflicting verification results can produce false positives and false negatives.
Loading 2506.14245v2…