PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang

arXiv:2504.13837v5cs.AIcs.CLcs.CV

TL;DR

The paper asks whether current RLVR discovers reasoning abilities beyond those already present in base models. It evaluates this question using large-k pass@k comparisons, coverage, perplexity, and algorithm-level analyses, finding that RLVR typically improves sampling efficiency without eliciting fundamentally new reasoning patterns.

  • Problem

    Despite RLVR’s empirical success, it remains unclear whether current methods acquire novel reasoning abilities through exploration or mainly use patterns already present in the base model.

  • Method

    The study compares base and RLVR-trained models across model families, sizes, benchmarks, and algorithms using pass@k, coverage, and perplexity analyses.

  • Results

    Current RLVR rarely elicits fundamentally new reasoning patterns: it improves sampling efficiency, while its reasoning capabilities remain bounded by the base model; distillation can expand them.

  • Takeaways & Limitations

    The findings indicate that current RLVR has not fully realized reinforcement learning’s potential to elicit novel reasoning abilities, motivating improved exploration and interaction paradigms.

  • Takeaways & Limitations

    The study is constrained by limited access to proprietary models and training pipelines, while emerging techniques may mitigate the identified limitations.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs), particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.

1. Introduction

This study examines whether current RLVR elicits reasoning abilities beyond the base model. Across broad evaluations, it finds that RLVR improves sampling efficiency but does not expand—and may narrow—the model’s reasoning coverage.

  • RLVR optimizes pretrained or chain-of-thought-finetuned models using automatically computable rewards tied to mathematical correctness or code unit tests.
  • The study uses pass@k across multiple LLM families, model sizes, benchmarks, and RLVR algorithms to compare base and RLVR-trained models.The metric captures performance when a problem receives multiple sampling attempts, rather than only average-case decoding behavior.
  • At small k, RLVR models outperform their base models, but base models surpass them as k increases across benchmarks and LLM families.This pattern indicates that RLVR can improve average sampling efficiency without expanding the set of solvable problems.
  • Coverage and perplexity analyses show that RLVR-generated reasoning paths already exist within the base model’s output distribution.RLVR improves pass@1 by sampling more efficiently on problems already solvable by the base model, rather than solving new problems.
  • Across algorithms including PPO, GRPO, and Reinforce++, the sampling-efficiency gap varies little but remains consistently large relative to the base model’s upper bound.The gap is defined using RL pass@1 and base-model pass@256 as a proxy for upper-bound performance.
  • Unlike RLVR, distillation can transfer new reasoning patterns from a stronger teacher and expand reasoning scope beyond the base model.
  • The findings expose a gap between current RLVR methods and reinforcement learning’s goal of discovering genuinely new reasoning strategies.The paper points to better exploration, continual data scaling, fine-grained process signals, and multi-turn agent interaction as improved paradigms.

2. Preliminaries

RLVR uses verifiable rewards and reinforcement-learning objectives to optimize language-model policies, while pass@k measures the boundary of reasoning capacity through repeated sampling. The setup compares base and RL-trained models under task-specific starting-model conventions.

  • Reinforcement Learning with Verifiable Rewards: RLVR assigns binary rewards when a mathematical answer is exactly correct or code passes unit tests, optionally adding format rewards for separated reasoning and final answers.The objective maximizes expected reward over prompts.
  • RLVR Algorithms: PPO maximizes a clipped surrogate objective using likelihood ratios and estimated advantages, with an optional KL term constraining deviation from the original policy.PPO and variants learn from on-policy samples generated by the current language model.
  • Experimental Setup: Zero-RL starts directly from pretrained base models for mathematics, whereas coding and visual reasoning compare instruction-tuned models with their RLVR-trained counterparts.The differing conventions reflect instability and limited effectiveness reported for pure zero-RL in coding and visual reasoning.
  • Metrics for LLM Reasoning Capacity Boundary: Pass@k counts a problem as solved when at least one of k sampled outputs passes verification, targeting reasoning potential rather than average-case performance.The metric is extended from code generation to all tasks with verifiable rewards.
  • Metrics for LLM Reasoning Capacity Boundary: For mathematical evaluation, incorrect chains of thought can accidentally produce correct answers as k increases, so the study manually checks chain-of-thought correctness on selected outputs.Coding pass@k is less vulnerable because passing unit tests is nearly impossible through guesswork.

3. RLVR’s Effect on Reasoning Capacity Boundary

The study evaluates base and RLVR models across mathematics, coding, and visual reasoning using extensive pass@k experiments. RLVR improves low-k sampling performance, but base models generally cover more solvable problems as k grows, including in coding and multimodal tasks.

  • Experimental Setup: The experiments span mathematics, code generation, and visual reasoning across multiple model families, sizes, benchmarks, and RLVR configurations.The setup uses consistent zero-shot or benchmark-default prompts for base and RLVR models.
  • Mathematical Reasoning: At k = 1, RL-trained models outperform base models, but base models catch up and surpass them as k increases; on Minerva with a 32B model, the base leads by approximately 9% at k = 128.The result indicates higher low-k sampling likelihood for RLVR and broader high-k coverage for the base model.
  • Mathematical Reasoning: An RL model initially performs nearly 30% above its base model before being surpassed as k increases, confirming the low-k advantage and narrower overall coverage pattern.The same conclusion is reported for models trained with Oat-Zero and DAPO.
  • Mathematical Reasoning: Manual analysis of difficult AIME24 problems finds that base-model samples include long, reflective correct chains of thought, demonstrating substantial reasoning ability in the original model.The inspected chains were selected from 2048 samplings of the hardest questions.
  • Code Generation: Across three coding benchmarks, RLVR produces pass@k trends highly consistent with mathematics, while unit-test verification makes pass@k relatively reliable against guesswork.The coding evaluation includes LiveCodeBench v5, HumanEval+, and MBPP+.
  • Visual Reasoning: Visual-reasoning results are highly consistent with mathematics and coding, suggesting that the original multimodal model covers a broader set of solvable questions.The experiments use Qwen2.5-VL-7B trained with EasyR1 on visual mathematical reasoning.
  • Validity of Chain-of-Thought: For challenging problems with average accuracy below 5%, both original and RL models have at least one correct chain of thought on 7 of 8 inspected problems.These inspections support the validity of the chains of thought used in the analysis.

4. Deep Analysis

The analysis finds that RLVR improves sampling efficiency but does not expand reasoning coverage beyond the base model. Across coverage, perplexity, algorithm, training, and temperature analyses, RLVR remains bounded by the base model, whereas distillation can expand reasoning capabilities.

  • 4.1. Reasoning Paths Already Present in Base Models: RLVR increases high-accuracy outcomes but also raises the frequency of unsolvable problems, indicating improved sampling efficiency without new solvable problems.The accuracy distribution shifts toward both accuracy near 1.0 and accuracy 0.
  • 4.1. Reasoning Paths Already Present in Base Models: The RL-trained model solves nearly a subset of the problems solvable by the base model, with many base-only successes and few RLVR-only successes.This pattern appears on AIME24, MATH500, and coding tasks.
  • 4.1. Reasoning Paths Already Present in Base Models: Base-model perplexity on RLVR responses closely matches the lower-perplexity portion of base-generated responses, supporting that RLVR paths already lie within the base distribution.As RL training progresses, PPLBase(YRL|x) decreases, consistent with sharpening the base prior rather than expanding beyond it.
  • 4.1. Reasoning Paths Already Present in Base Models: The combined analyses conclude that RLVR improves average scores through more efficient sampling, while narrowing coverage and leaving reasoning capacity bounded by the base model.The paper reports no fundamentally new reasoning capabilities from current RLVR training.
  • 4.2. Distillation: Distillation is presented as an alternative that can introduce teacher-generated reasoning patterns and expand the reasoning ability of smaller base models.Its training data consist of long chain-of-thought traces generated by a powerful reasoning teacher.
  • 4.3. Comparing Different RL Algorithms: Across six RL algorithms, the Sampling Efficiency Gap ranges from 43.9 for GRPO to 42.6 for RLOO and remains above 40 points on the in-domain test set.Lower ΔSE is better, and the results indicate that existing methods remain far from optimal sampling efficiency.
  • 4.4. Effects of RL Training: Training increases pass@1 from 26.1 to 42.5 while progressively decreasing pass@256, indicating a reduced reasoning boundary as RLVR training continues.This asymptotic effect shows higher single-sample performance alongside lower large-k performance.
  • 4.5. Effects of Number of Rollouts: Increasing rollouts from n = 8 to 32 improves pass@k slightly, but the RL-trained model is still eventually outperformed by the base model.Whether scaling RLVR can eventually surpass the base model remains open.

5. Discussion

The discussion attributes current RLVR limitations to inefficient exploration in language models’ vast action space and reliance on pretrained priors and binary rewards. It proposes higher-level exploration and curriculum-based scaling as possible remedies.

  • Discussion 1: Key Differences Between Traditional RL and RLVR for LLMs are Vast Action Space and Pretrained Priors: Language models face an exponentially larger action space than traditional RL agents, making direct token-level exploration difficult.The discussion contrasts this setting with traditional RL systems such as AlphaGo Zero and DQN.
  • Discussion 2: Priors as a Double-Edged Sword in This Vast Action Space: Pretrained priors constrain naive sampling exploration, because responses deviating from the prior are highly likely to become invalid or nonsensical.This limits discovery of reasoning patterns beyond the prior.
  • Possible Future Work: The paper identifies inefficient exploration in a vast action space and binary outcome rewards as possible root causes of current RLVR limitations.These factors motivate proposed changes to exploration and training data.
  • Possible Future Work: High-level exploration in program-level abstraction spaces is proposed to help discover out-of-prior reasoning patterns and previously unseen knowledge structures.AlphaEvolve is given as an example of self-evolution at a higher abstraction level.
  • Possible Future Work: Curriculum-based data scaling may reduce exploration difficulty by improving success on easier subproblems before training on harder parent tasks.The paper says realizing this potential requires a deliberate, large-scale data-RL iteration pipeline.

6. Related Work

Related work finds that the fundamental impact of RLVR on reasoning remains underexplored. Prior studies suggest reflective behaviors originate in base models, while other work reports declining pass@k after RLVR training.

  • Analysis of RLVR: Several studies suggest that reflective behaviors in RLVR models originate from base models rather than being learned through reinforcement learning.This evidence aligns with the paper’s investigation of whether RLVR introduces novel reasoning patterns.
  • Analysis of RLVR: Dang et al. observed a decline in pass@k performance after RLVR training, contributing to concerns about RLVR’s effect on reasoning boundaries.The related-work discussion presents this as prior evidence rather than a new result of the present study.

7. Conclusion and Limitations

Current RLVR methods improve sampling efficiency but rarely elicit reasoning patterns beyond the base model, leaving reinforcement learning’s exploration potential unrealized. The study’s conclusions are limited by proprietary models and rapidly evolving RL techniques.

  • Conclusion: Current RLVR-trained models’ reasoning capabilities remain bounded by those of their base models.The authors attribute this limitation partly to ineffective exploration in the vast language space.
  • Conclusion: Improved RL paradigms may require better exploration, continual data scaling, fine-grained process signals, and multi-turn agent interaction.These directions are proposed to address the gap between existing RLVR methods and genuinely new reasoning strategies.
  • Limitations: The analysis is constrained because many capable models and training pipelines remain proprietary.The authors also note that rapidly emerging techniques may mitigate some identified limitations.

A. Implementation Details

The implementation estimates pass@k with many samples per problem to reduce variance, and uses benchmark-dependent maximum sample counts. The appendix also describes critic-free advantage estimators used to reduce computational overhead.

  • A.1. RLVR Algorithms: GRPO normalizes each response reward within a group, while RLOO uses a leave-one-out baseline within each batch.For GRPO, the advantage is based on group reward mean and standard deviation; RLOO subtracts the mean of other batch rewards.
  • A.2. Low-Variance pass@k Estimation: The unbiased pass@k estimator samples n outputs per problem, counts correct outputs as c_i, and estimates performance across the evaluation dataset.The method requires n ≥ k.
  • A.2. Low-Variance pass@k Estimation: The estimator enables low-variance pass@k estimates for every k ≤ n.This avoids relying on only k samples per problem, which can produce high variance.
  • A.2. Low-Variance pass@k Estimation: The experiments set n to the largest k shown in each pass@k curve, typically 128, 256, or 1024.The chosen n varies by benchmark and, for Olympiad, by model family because of differences in base-model capacity.

B. More Related Works

Related work covers RL-based reasoning models, recent analyses of RLVR’s effects, and supplementary evaluations across math and coding benchmarks. The paper positions its contribution as a systematic demonstration that reasoning paths are already embedded in base models.

  • Reinforcement Learning for LLM Reasoning: RLVR has become a prominent post-training approach for improving mathematical and programming reasoning with verifiable rewards.The related-work discussion places RLVR alongside supervised fine-tuning, self-improvement, and reward-model-based reinforcement learning.
  • Analysis of RLVR: Prior analyses highlighted reflective behaviors in R1-like models but did not examine the relationship between base and RL models.This paper extends that line of inquiry beyond reflective behavior to all reasoning paths.
  • Analysis of RLVR: The paper reports that reflective behaviors and all analyzed reasoning paths are already embedded in the base model.It therefore concludes that RLVR does not elicit new reasoning abilities beyond the base model.
  • Additional Evaluations: Supplementary figures report results for SimpleRLZoo, Oat-Zero-7B, DAPO-32B, Coder-R1, and SimpleRLZoo-7B across math and coding benchmarks.The cited figures cover GSM8K, AMC23, AIME24, LiveCodeBench, and filtered AIME24.
  • CoT Case Analysis: AIME24 evaluation uses a heuristic filter that removes easily guessable problems before manual inspection of correct chain-of-thought responses.Applying the filter to 30 AIME24 questions yields 18 retained problems.

C.4. Perplexity Analysis

Perplexity and algorithm analyses examine how RLVR changes output distributions and how rollout choices affect pass@k. The reported evidence emphasizes distribution sharpening, algorithmic trade-offs, and higher-k gains from larger rollouts.

  • C.4. Perplexity Analysis: RL training gradually decreases PPL_Base(Y_RL|x), indicating that RLVR sharpens the distribution within the base model’s output space.The analysis compares early, middle, and final RL checkpoints using 32 sampled responses per problem.
  • Additional Results: Figures and tables provide additional pass@k curves, perplexity evolution, and detailed pass@1 and pass@256 values across algorithms and training steps.These materials supplement the main comparisons with pointwise metric values and training-step analyses.
  • RL Algorithm Comparisons: DAPO achieves slightly higher pass@1 across three datasets but uses approximately 3∼6× more samples per training batch and drops significantly at k = 256.RLOO and Reinforce++ perform consistently across k = 1 to 256 while maintaining efficient training costs.
  • Ablation Study: Increasing rollout number n from 8 to 32 raises pass@128 despite lower pass@1 under a shortened, unconverged training run.The ablation keeps prompt batch size unchanged, increasing computation per step, and trains for only 220 steps.

C.7. Solvable Problem Coverage Analysis

The coverage analysis compares which problems the base and RLVR models solve across sampled responses. It finds that base-only successes are common, while RLVR-only successes are rare, with supporting solvable-problem indices reported for AIME24 and LiveCodeBench.

  • Coverage comparison: Many problems are solved by the base model but not by RLVR, whereas RLVR-only successes are very rare.In MATH500, RLVR-only cases comprise about 1%, or approximately five problems.
  • AIME24: Most AIME24 problems solved by the RL model are also solvable by the base model.
  • LiveCodeBench: The analysis lists solvable-problem indices for LiveCodeBench problems 400–450.

C.8. Temperature and Entropy Analysis

The temperature and entropy analysis evaluates how sampling settings affect base and RL model reasoning performance. It uses T = 0.6 for the main experiments and separately matches RLVR output entropy to the base model for fair comparison.

  • Temperature effects: T = 0.6 is used in the main experiments because it supports the best reasoning performance of both models.The base model’s performance drops above temperature 1.0, while the RL model remains relatively stable across temperatures.
  • Entropy matching: RLVR temperature is increased to approximately match the base model’s output entropy for fair comparison.On AMC23, the base model uses T = 0.6 with Ebase = 0.51, while the RLVR model uses T = 0.9 to achieve ERL = 0.47.
  • Training dynamics: Figure 19 tracks training reward, response length, and generation entropy over training.
Loading 2504.13837v5…